[
https://issues.apache.org/jira/browse/HADOOP-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980402#action_12980402
]
Noah Watkins commented on HADOOP-7099:
--------------------------------------
Yes, this is for Ceph. The solution we are using is great: code is identical to
RawLocalFileSystem, but for files located under a Ceph mount point,
getBlockLocations consults the in-kernel client via an IOCTL call written in
JNI. That's the high-level view. In practice it has been a little hard to get
right in terms of something that fits well into the Hadoop code base.
So, the fundamental issue is that access to Ceph is through the local file
system (i.e. file:///), but for a subset of the file system hierarchy (i.e. the
Ceph mount point), a different file system implementation should be used, not
just for accounting, but also for altered functionality (i.e. the ioctl
availability for Ceph files). We use a configuration parameter to specify the
mount point, and force access through the Hadoop Ceph File System to stay
within the mount point.
With sub-classing, the solution is nice because changing the URI (ceph:///)
doesn't cause the problem I cited in the previous comment (checkPath blowing
up).
Does that paint a clearer picture? We now have our code working well, but are
struggling with how to package it up for distribution.
Thanks!
> Make RawLocalFileSystem more friendly to sub-classing
> -----------------------------------------------------
>
> Key: HADOOP-7099
> URL: https://issues.apache.org/jira/browse/HADOOP-7099
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
> Affects Versions: 0.23.0
> Reporter: Noah Watkins
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HADOOP-7099.patch
>
>
> This patch does 3 things that makes sub-classing RawLocalFileSystem easier.
> First, it adds a constructor that allows a sub-class to avoid calling
> getInitialWorkingDirectory(). This is important because if a sub-class has an
> initially null uri (prior to initialize() being called), then
> getInitialWorkingDirectory() will cause a NullPointerException when it tries
> to work with the FS's uri.
> Second, allows subclasses to modify the working directory.
> The third thing this patch does is change loadPermissions to not pass the URI
> object to the File(URI) constructor, but rather pass the string
> representation of the path. This is important because URI's that are not
> using the "file" scheme will cause the File(URI) constructor to throw an
> exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
