pitrou commented on code in PR #41564:
URL: https://github.com/apache/arrow/pull/41564#discussion_r1603496580
##########
cpp/src/arrow/filesystem/s3fs.h:
##########
@@ -166,6 +166,13 @@ struct ARROW_EXPORT S3Options {
/// Whether to allow deletion of buckets
bool allow_bucket_deletion = false;
+ /// Whether to sanitize that a bucket can be written to when calling
OpenOutputStream.
+ /// Disabling this will lead to less S3 requests issued and is therefore
relevant
+ /// for latency-sensitive applications, at the cost of the OutputStream may
throwing an
+ /// exception at a later stage (i.e. at writing or closing) if e.g. the
bucket does not
+ /// exist.
+ bool sanitize_bucket_on_open = true;
Review Comment:
Perhaps we should make this more general, to open up other potential
optimizations?
```c++
/// Whether to allow file-open methods to return before the actual open
///
/// Enabling this true may reduce the latency of `OpenInputStream`,
`OpenOutpuStream`,
/// and similar methods, by reducing the number of roundtrips necessary.
It may also
/// allow usage of more efficient S3 APIs for small files.
/// The downside is that failure conditions such as attempting to open a
file in a
/// non-existing bucket will only be reported when actual I/O is done (at
worse,
/// when attempting to close the file).
bool allow_delayed_open = false;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
