[
https://issues.apache.org/jira/browse/HADOOP-17682?focusedWorklogId=647648&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-647648
]
ASF GitHub Bot logged work on HADOOP-17682:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 08/Sep/21 00:37
Start Date: 08/Sep/21 00:37
Worklog Time Spent: 10m
Work Description: sumangala-patki commented on a change in pull request
#2975:
URL: https://github.com/apache/hadoop/pull/2975#discussion_r703940807
##########
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##########
@@ -669,44 +672,64 @@ public void createDirectory(final Path path, final
FsPermission permission,
public AbfsInputStream openFileForRead(final Path path,
final FileSystem.Statistics statistics, TracingContext tracingContext)
- throws AzureBlobFileSystemException {
- return openFileForRead(path, Optional.empty(), statistics, tracingContext);
+ throws IOException {
+ return openFileForRead(path, Optional.empty(), statistics,
+ tracingContext);
}
- public AbfsInputStream openFileForRead(final Path path,
- final Optional<Configuration> options,
+ public AbfsInputStream openFileForRead(Path path,
+ final Optional<OpenFileParameters> parameters,
final FileSystem.Statistics statistics, TracingContext tracingContext)
- throws AzureBlobFileSystemException {
- try (AbfsPerfInfo perfInfo = startTracking("openFileForRead",
"getPathStatus")) {
+ throws IOException {
+ try (AbfsPerfInfo perfInfo = startTracking("openFileForRead",
+ "getPathStatus")) {
LOG.debug("openFileForRead filesystem: {} path: {}",
- client.getFileSystem(),
- path);
+ client.getFileSystem(), path);
+ FileStatus fileStatus = parameters.map(OpenFileParameters::getStatus)
+ .orElse(null);
String relativePath = getRelativePath(path);
-
- final AbfsRestOperation op = client
- .getPathStatus(relativePath, false, tracingContext);
- perfInfo.registerResult(op.getResult());
-
- final String resourceType =
op.getResult().getResponseHeader(HttpHeaderConfigurations.X_MS_RESOURCE_TYPE);
- final long contentLength =
Long.parseLong(op.getResult().getResponseHeader(HttpHeaderConfigurations.CONTENT_LENGTH));
- final String eTag =
op.getResult().getResponseHeader(HttpHeaderConfigurations.ETAG);
+ String resourceType, eTag;
+ long contentLength;
+ if (fileStatus instanceof VersionedFileStatus) {
+ path = path.makeQualified(this.uri, path);
+ Preconditions.checkArgument(fileStatus.getPath().equals(path),
+ String.format(
+ "Filestatus path [%s] does not match with given path [%s]",
+ fileStatus.getPath(), path));
+ resourceType = fileStatus.isFile() ? FILE : DIRECTORY;
+ contentLength = fileStatus.getLen();
+ eTag = ((VersionedFileStatus) fileStatus).getVersion();
+ } else {
+ if (fileStatus != null) {
+ LOG.warn(
+ "Fallback to getPathStatus REST call as provided filestatus "
+ + "is not of type VersionedFileStatus");
+ }
+ AbfsHttpOperation op = client.getPathStatus(relativePath, false,
+ tracingContext).getResult();
+ resourceType = op.getResponseHeader(
+ HttpHeaderConfigurations.X_MS_RESOURCE_TYPE);
+ contentLength = Long.parseLong(
+ op.getResponseHeader(HttpHeaderConfigurations.CONTENT_LENGTH));
+ eTag = op.getResponseHeader(HttpHeaderConfigurations.ETAG);
+ }
if (parseIsDirectory(resourceType)) {
throw new AbfsRestOperationException(
Review comment:
yes, there is a contract test to verify that a FileNotFound exception is
thrown on attempt to read a directory
org.apache.hadoop.fs.contract.AbstractContractOpenTest#testOpenReadDir
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 647648)
Time Spent: 8h (was: 7h 50m)
> ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters
> ------------------------------------------------------------------------------
>
> Key: HADOOP-17682
> URL: https://issues.apache.org/jira/browse/HADOOP-17682
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Reporter: Sumangala Patki
> Assignee: Sumangala Patki
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 8h
> Remaining Estimate: 0h
>
> ABFS open methods require certain information (contentLength, eTag, etc) to
> to create an InputStream for the file at the given path. This information is
> retrieved via a GetFileStatus request to backend.
> However, client applications may often have access to the FileStatus prior to
> invoking the open API. Providing this FileStatus to the driver through the
> OpenFileParameters argument of openFileWithOptions() can help avoid the call
> to Store for FileStatus.
> This PR adds handling for the FileStatus instance (if any) provided via the
> OpenFileParameters argument.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
