LakshSingla commented on code in PR #14720:
URL: https://github.com/apache/druid/pull/14720#discussion_r1280451339
##########
docs/multi-stage-query/reference.md:
##########
@@ -389,13 +389,13 @@ The following common service properties control how
durable storage behaves:
|Parameter |Default | Description
|
|-------------------|----------------------------------------|----------------------|
-|`druid.msq.intermediate.storage.bucket` | n/a | The bucket in S3 where you
want to store intermediate files. |
-|`druid.msq.intermediate.storage.chunkSize` | 100MiB | Optional. Defines the
size of each chunk to temporarily store in
`druid.msq.intermediate.storage.tempDir`. The chunk size must be between 5 MiB
and 5 GiB. A large chunk size reduces the API calls made to the durable
storage, however it requires more disk space to store the temporary chunks.
Druid uses a default of 100MiB if the value is not provided.|
|`druid.msq.intermediate.storage.enable` | true | Required. Whether to enable
durable storage for the cluster.|
-|`druid.msq.intermediate.storage.maxRetry` | 10 | Optional. Defines the max
number times to attempt S3 API calls to avoid failures due to transient errors.
|
+|`druid.msq.intermediate.storage.type` | `s3` if your deep storage is S3 |
Required. The type of storage to use. Currently only `s3` is supported. |
+|`druid.msq.intermediate.storage.chunkSize` | 100MiB | Optional. Defines the
size of each chunk to temporarily store in
`druid.msq.intermediate.storage.tempDir`. The chunk size must be between 5 MiB
and 5 GiB. A large chunk size reduces the API calls made to the durable
storage, however it requires more disk space to store the temporary chunks.
Druid uses a default of 100MiB if the value is not provided.|
+|`druid.msq.intermediate.storage.maxRetry` | 10 | Optional. Defines the max
number times to attempt S3 API calls to avoid failures due to transient errors.
|
+|`druid.msq.intermediate.storage.bucket` | n/a | The bucket in S3 where you
want to store intermediate files. |
Review Comment:
I think we should do it in the order like: type > bucket > prefix > tempDir
> maxRetry > chunkSize.
First three are required. Max retry and chunk size is something that we
least expect the users to set, with chunk size being the least of these two. I
am of the opinion that we can also undocument it.
##########
docs/multi-stage-query/reference.md:
##########
@@ -389,13 +389,13 @@ The following common service properties control how
durable storage behaves:
|Parameter |Default | Description
|
|-------------------|----------------------------------------|----------------------|
-|`druid.msq.intermediate.storage.bucket` | n/a | The bucket in S3 where you
want to store intermediate files. |
-|`druid.msq.intermediate.storage.chunkSize` | 100MiB | Optional. Defines the
size of each chunk to temporarily store in
`druid.msq.intermediate.storage.tempDir`. The chunk size must be between 5 MiB
and 5 GiB. A large chunk size reduces the API calls made to the durable
storage, however it requires more disk space to store the temporary chunks.
Druid uses a default of 100MiB if the value is not provided.|
|`druid.msq.intermediate.storage.enable` | true | Required. Whether to enable
durable storage for the cluster.|
-|`druid.msq.intermediate.storage.maxRetry` | 10 | Optional. Defines the max
number times to attempt S3 API calls to avoid failures due to transient errors.
|
+|`druid.msq.intermediate.storage.type` | `s3` if your deep storage is S3 |
Required. The type of storage to use. Currently only `s3` is supported. |
+|`druid.msq.intermediate.storage.chunkSize` | 100MiB | Optional. Defines the
size of each chunk to temporarily store in
`druid.msq.intermediate.storage.tempDir`. The chunk size must be between 5 MiB
and 5 GiB. A large chunk size reduces the API calls made to the durable
storage, however it requires more disk space to store the temporary chunks.
Druid uses a default of 100MiB if the value is not provided.|
+|`druid.msq.intermediate.storage.maxRetry` | 10 | Optional. Defines the max
number times to attempt S3 API calls to avoid failures due to transient errors.
|
+|`druid.msq.intermediate.storage.bucket` | n/a | The bucket in S3 where you
want to store intermediate files. |
|`druid.msq.intermediate.storage.prefix` | n/a | S3 prefix to store
intermediate stage results. Provide a unique value for the prefix. Don't share
the same prefix between clusters. If the location includes other files or
directories, then they will get cleaned up as well. |
|`druid.msq.intermediate.storage.tempDir`| n/a | Required. Directory path on
the local disk to temporarily store intermediate stage results. |
Review Comment:
Is this duplicated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
