317brian commented on code in PR #16051:
URL: https://github.com/apache/druid/pull/16051#discussion_r1523793377
##########
docs/multi-stage-query/reference.md:
##########
@@ -149,6 +149,39 @@ The following runtime parameters must be configured to
export into an S3 destina
| `druid.export.storage.s3.maxRetry` | No | Defines the max
number times to attempt S3 API calls to avoid failures due to transient errors.
| 10 |
| `druid.export.storage.s3.chunkSize` | No | Defines the size
of each chunk to temporarily store in `tempDir`. The chunk size must be between
5 MiB and 5 GiB. A large chunk size reduces the API calls to S3, however it
requires more disk space to store the temporary chunks. | 100MiB |
+
+##### GS
+
+Export results to GCS by passing the function `google()` as an argument to the
`EXTERN` function. Note that this requires the `druid-google-extensions`.
+The `google()` function is a Druid function that configures the connection.
Arguments for `google()` should be passed as named parameters with the value in
single quotes like the following example:
+
+```sql
+INSERT INTO
+ EXTERN(
+ google(bucket => 'your_bucket', prefix => 'prefix/to/files')
+ )
+AS CSV
+SELECT
+ <column>
+FROM <table>
+```
+
+Supported arguments for the function:
+
+| Parameter | Required | Description
| Default |
+|-------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|
+| `bucket` | Yes | The GS bucket to which the files are exported to.
The bucket and prefix combination should be whitelisted in
`druid.export.storage.google.allowedExportPaths`.
| n/a |
+| `prefix` | Yes | Path where the exported files would be created. The
export query expects the destination to be empty. If the location includes
other files, then the query will fail. The bucket and prefix combination should
be whitelisted in `druid.export.storage.google.allowedExportPaths`. | n/a |
+
+The following runtime parameters must be configured to export into an S3
destination:
Review Comment:
```suggestion
The following runtime parameters must be configured to export into an GS
destination:
```
##########
docs/multi-stage-query/reference.md:
##########
@@ -149,6 +149,39 @@ The following runtime parameters must be configured to
export into an S3 destina
| `druid.export.storage.s3.maxRetry` | No | Defines the max
number times to attempt S3 API calls to avoid failures due to transient errors.
| 10 |
| `druid.export.storage.s3.chunkSize` | No | Defines the size
of each chunk to temporarily store in `tempDir`. The chunk size must be between
5 MiB and 5 GiB. A large chunk size reduces the API calls to S3, however it
requires more disk space to store the temporary chunks. | 100MiB |
+
+##### GS
+
+Export results to GCS by passing the function `google()` as an argument to the
`EXTERN` function. Note that this requires the `druid-google-extensions`.
+The `google()` function is a Druid function that configures the connection.
Arguments for `google()` should be passed as named parameters with the value in
single quotes like the following example:
+
+```sql
+INSERT INTO
+ EXTERN(
+ google(bucket => 'your_bucket', prefix => 'prefix/to/files')
+ )
+AS CSV
+SELECT
+ <column>
+FROM <table>
+```
+
+Supported arguments for the function:
+
+| Parameter | Required | Description
| Default |
+|-------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|
+| `bucket` | Yes | The GS bucket to which the files are exported to.
The bucket and prefix combination should be whitelisted in
`druid.export.storage.google.allowedExportPaths`.
| n/a |
+| `prefix` | Yes | Path where the exported files would be created. The
export query expects the destination to be empty. If the location includes
other files, then the query will fail. The bucket and prefix combination should
be whitelisted in `druid.export.storage.google.allowedExportPaths`. | n/a |
+
+The following runtime parameters must be configured to export into an S3
destination:
+
+| Runtime Parameter | Required | Description
| Default |
+|--------------------------------------------------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|
+| `druid.export.storage.google.tempLocalDir` | Yes | Directory used
on the local storage of the worker to store temporary files required while
uploading the data.
| n/a |
+| `druid.export.storage.google.allowedExportPaths` | Yes | An array of GS
prefixes that are whitelisted as export destinations. Export queries fail if
the export destination does not match any of the configured prefixes. Example:
`[\"gs://bucket1/export/\", \"gs://bucket2/export/\"]` | n/a |
Review Comment:
```suggestion
| `druid.export.storage.google.allowedExportPaths` | Yes | An array of
GS prefixes that are allowed as export destinations. Export queries fail if the
export destination does not match any of the configured prefixes. Example:
`[\"gs://bucket1/export/\", \"gs://bucket2/export/\"]` | n/a |
```
##########
docs/multi-stage-query/reference.md:
##########
@@ -149,6 +149,39 @@ The following runtime parameters must be configured to
export into an S3 destina
| `druid.export.storage.s3.maxRetry` | No | Defines the max
number times to attempt S3 API calls to avoid failures due to transient errors.
| 10 |
| `druid.export.storage.s3.chunkSize` | No | Defines the size
of each chunk to temporarily store in `tempDir`. The chunk size must be between
5 MiB and 5 GiB. A large chunk size reduces the API calls to S3, however it
requires more disk space to store the temporary chunks. | 100MiB |
+
+##### GS
+
+Export results to GCS by passing the function `google()` as an argument to the
`EXTERN` function. Note that this requires the `druid-google-extensions`.
+The `google()` function is a Druid function that configures the connection.
Arguments for `google()` should be passed as named parameters with the value in
single quotes like the following example:
+
+```sql
+INSERT INTO
+ EXTERN(
+ google(bucket => 'your_bucket', prefix => 'prefix/to/files')
+ )
+AS CSV
+SELECT
+ <column>
+FROM <table>
+```
+
+Supported arguments for the function:
+
+| Parameter | Required | Description
| Default |
+|-------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|
+| `bucket` | Yes | The GS bucket to which the files are exported to.
The bucket and prefix combination should be whitelisted in
`druid.export.storage.google.allowedExportPaths`.
| n/a |
+| `prefix` | Yes | Path where the exported files would be created. The
export query expects the destination to be empty. If the location includes
other files, then the query will fail. The bucket and prefix combination should
be whitelisted in `druid.export.storage.google.allowedExportPaths`. | n/a |
+
+The following runtime parameters must be configured to export into an S3
destination:
+
+| Runtime Parameter | Required | Description
| Default |
+|--------------------------------------------------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|
+| `druid.export.storage.google.tempLocalDir` | Yes | Directory used
on the local storage of the worker to store temporary files required while
uploading the data.
| n/a |
+| `druid.export.storage.google.allowedExportPaths` | Yes | An array of GS
prefixes that are whitelisted as export destinations. Export queries fail if
the export destination does not match any of the configured prefixes. Example:
`[\"gs://bucket1/export/\", \"gs://bucket2/export/\"]` | n/a |
+| `druid.export.storage.google.maxRetry` | No | Defines the
max number times to attempt GS API calls to avoid failures due to transient
errors.
| 10 |
+| `druid.export.storage.google.chunkSize` | No | Defines the
size of each chunk to temporarily store in `tempDir`. A large chunk size
reduces the API calls to GS, however it requires more disk space to store the
temporary chunks. | 4MiB |
Review Comment:
```suggestion
| `druid.export.storage.google.chunkSize` | No | Defines the
size of each chunk to temporarily store in `tempDir`. A large chunk size
reduces the API calls to GS; however, it requires more disk space to store the
temporary chunks. | 4MiB |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
