This is an automated email from the ASF dual-hosted git repository.
felixybw pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 9f424a1d0 [VL] Add support for GCS retry properties (#5858)
9f424a1d0 is described below
commit 9f424a1d05180fcbd00ea9b84a8d3d8b7dcca8c0
Author: Sandino Flores <[email protected]>
AuthorDate: Fri May 24 15:19:34 2024 -0600
[VL] Add support for GCS retry properties (#5858)
The Velox GCS connector was recently added a couple of properties:
HiveConfig::kGCSMaxRetryCount: Retry until a number of transient errors is
detected.
HiveConfig::kGCSMaxRetryTime: Retry until a time elapses.
This properties are useful to allow a client to either fail immediately or
keep retring for several minutes.
If none of them is set, then it will use the defaults set by GCS.
This change was tested manually, by entering an unnaccesible GCS bucket
then verifying that the properties are honored.
---
cpp/velox/utils/ConfigExtractor.cc | 13 +++++++++++++
docs/get-started/VeloxGCS.md | 14 ++++++++++++++
2 files changed, 27 insertions(+)
diff --git a/cpp/velox/utils/ConfigExtractor.cc
b/cpp/velox/utils/ConfigExtractor.cc
index 0cbba37a7..a71f14322 100644
--- a/cpp/velox/utils/ConfigExtractor.cc
+++ b/cpp/velox/utils/ConfigExtractor.cc
@@ -121,6 +121,19 @@ std::shared_ptr<facebook::velox::core::MemConfig>
getHiveConfig(std::shared_ptr<
}
}
+ //
https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/master/gcs/CONFIGURATION.md#http-transport-configuration
+ //
https://cloud.google.com/cpp/docs/reference/storage/latest/classgoogle_1_1cloud_1_1storage_1_1LimitedErrorCountRetryPolicy
+ auto gsMaxRetryCount = conf->get("spark.hadoop.fs.gs.http.max.retry");
+ if (gsMaxRetryCount.hasValue()) {
+
hiveConfMap[facebook::velox::connector::hive::HiveConfig::kGCSMaxRetryCount] =
gsMaxRetryCount.value();
+ }
+
+ //
https://cloud.google.com/cpp/docs/reference/storage/latest/classgoogle_1_1cloud_1_1storage_1_1LimitedTimeRetryPolicy
+ auto gsMaxRetryTime = conf->get("spark.hadoop.fs.gs.http.max.retry-time");
+ if (gsMaxRetryTime.hasValue()) {
+
hiveConfMap[facebook::velox::connector::hive::HiveConfig::kGCSMaxRetryTime] =
gsMaxRetryTime.value();
+ }
+
//
https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/master/gcs/CONFIGURATION.md#authentication
auto gsAuthType = conf->get("spark.hadoop.fs.gs.auth.type");
if (gsAuthType.hasValue()) {
diff --git a/docs/get-started/VeloxGCS.md b/docs/get-started/VeloxGCS.md
index d87d9b960..453bee117 100644
--- a/docs/get-started/VeloxGCS.md
+++ b/docs/get-started/VeloxGCS.md
@@ -36,4 +36,18 @@ spark.hadoop.fs.gs.auth.service.account.json.keyfile // path
to the json file wi
For cases when a GCS mock is used, an optional endpoint can be provided:
```sh
spark.hadoop.fs.gs.storage.root.url // url to the mock gcs service including
starting with http or https
+```
+
+## Configuring GCS max retry count
+
+For cases when a transient server error is detected, GCS can be configured to
keep retrying until a number of transient error is detected.
+```sh
+spark.hadoop.fs.gs.http.max.retry // number of times to keep retrying unless a
non-transient error is detected
+```
+
+## Configuring GCS max retry time
+
+For cases when a transient server error is detected, GCS can be configured to
keep retrying until the retry loop exceeds a prescribed duration.
+```sh
+spark.hadoop.fs.gs.http.max.retry-time // a string representing the time keep
retring (10s, 1m, etc).
```
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
