Steve Loughran created HADOOP-16844:
---------------------------------------
Summary: FS/FC createFile() builder to add a lazy-check option for
async probes of the store
Key: HADOOP-16844
URL: https://issues.apache.org/jira/browse/HADOOP-16844
Project: Hadoop Common
Issue Type: Improvement
Components: fs, fs/azure, fs/s3
Affects Versions: 3.3.0
Reporter: Steve Loughran
For object stores, file creation can triggers checks for: path being to a file,
path not being a directory, parent path existing.
These are all done before an output stream is returned -even though nothing may
be manifest in the remote store until the screen is flashed or closed.
If we supported lazy checks we could do those probes against the store while
the application was actually generating data to write.
Changing the behaviour of open() would be too fundamental. What we can do is
add a new option to the createFile() builder which stores can support. When
set, the caller is declaring they are happy with late reporting of failures
-with any problems surfacing during a write/flush/close. It is notable with S3A
permissions issues only surface later on anyway.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
