[
https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156841#comment-16156841
]
Steve Loughran commented on HADOOP-14600:
-----------------------------------------
bq. I got lots of cuts and blood with getting mvn test -Dtest=foo running on
Window
I like that phrase, I'll have to use it.
If tests are timing out, that's something a (separate) patch should address.
h3. RawLocalFileSystem
L678, 687, 696, 751: need to fallback to non-native code if not present; a lot
of client uses of the hadoop libs don't use the native libs (including all our
tests)
we may want to think about actually falling back to the original mechanism if
the native IO Fails, e.g
{code}
if (!isPermissionLoaded() && nativeLibPresent) {
try {
loadPermissionsByNativeIO()
} catch (NativeIOException ex) {
LOG.debug("Native call failed", ex)
}
}
if (!isPermissionLoaded()) {
// invoked if
loadPermissionInfo();
}
{code}
Probably best to pull this out into some common method
"maybeLoadPermissionInfo".
Looking at the code, I see we don't appear to be doing any synchronization.
Really, if a call
is made to get the perm info, it should block other attempts. Making the
proposed
{{maybeLoadPermissionInfo()}} synchronized would be the easy way to do this
L751
don't bother with StringifyException. I know a lot of old code uses it, but its
best just to go
throw new RuntimeException(" failure in ..." + e, e)
this adds e.toString() to the exception text, and includes the nested exception
there for code and logs to print.
h3. NativeIO.java
L451: include the caught exception in the new one
L456. given this adds nothing, just rethrow the caught exception
h3. NativeIO.c
L435. Prefer {{if (path == NULL)}} for consistency.
h3. proposed tests:
# pass in a null path
# query a nonexistent file
> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the
> permissions
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-14600
> URL: https://issues.apache.org/jira/browse/HADOOP-14600
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.7.3
> Environment: file:// in a dir with many files
> Reporter: Steve Loughran
> Assignee: Ping Liu
> Attachments: HADOOP-14600.001.patch,
> TestRawLocalFileSystemContract.java
>
>
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws
> against the local FS, because {{FileStatus.getPemissions}} call forces
> {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI
> values.
> That is: for every other FS, what's a field lookup or even a no-op, on the
> local FS it's a process exec/spawn, with all the costs. This gets expensive
> if you have many files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
