Re: Private, LimitedPrivate and contrib modules
Sorry for the late reply .. I missed in in my inbox. On Sep 18, 2009, at 1:29 PM, Tom White wrote: I'm trying to better understand the meaning of the annotations defined in org.apache.hadoop.classification.InterfaceAudience. 1. Private is documented as being Intended for use only within Hadoop itself. Does this mean the whole Hadoop project, or the subproject that the annotated element is in? (Or another scope?) Private means private to the subproject (ie private to HDFS, to MR, etc). We should update the doc to clarify this. 2. Is a contrib module considered to be a part of the subproject? For example, if something is marked as LimitedPrivate in Common say, with intended audience MapReduce, can MapReduce contrib modules use it? I believe the answer should be no. Or, assuming the second meaning for 1, if something is marked as Private in MapReduce, can MapReduce contrib modules use it? Again I believe the answer should be no. Any concrete examples that we can use to examine/discuss this further Thanks, Tom
Re: Private, LimitedPrivate and contrib modules
Hi sanjay, Most of the contrib modules are in the same package as their containers. For example, the fair share scheduler is in contrib but its package name is org.apache.hadoop.mapred. Doesn't this mean that the fair-share scheduler code can use LimitedPrivate methods from org.apache.hadoop.mapred package? thanks, dhruba On Thu, Nov 5, 2009 at 11:05 AM, Sanjay Radia sra...@yahoo-inc.com wrote: Sorry for the late reply .. I missed in in my inbox. On Sep 18, 2009, at 1:29 PM, Tom White wrote: I'm trying to better understand the meaning of the annotations defined in org.apache.hadoop.classification.InterfaceAudience. 1. Private is documented as being Intended for use only within Hadoop itself. Does this mean the whole Hadoop project, or the subproject that the annotated element is in? (Or another scope?) Private means private to the subproject (ie private to HDFS, to MR, etc). We should update the doc to clarify this. 2. Is a contrib module considered to be a part of the subproject? For example, if something is marked as LimitedPrivate in Common say, with intended audience MapReduce, can MapReduce contrib modules use it? I believe the answer should be no. Or, assuming the second meaning for 1, if something is marked as Private in MapReduce, can MapReduce contrib modules use it? Again I believe the answer should be no. Any concrete examples that we can use to examine/discuss this further Thanks, Tom -- Connect to me at http://www.facebook.com/dhruba
[jira] Created: (HADOOP-6365) distributed cache doesn't work with HDFS and another file system
distributed cache doesn't work with HDFS and another file system Key: HADOOP-6365 URL: https://issues.apache.org/jira/browse/HADOOP-6365 Project: Hadoop Common Issue Type: Bug Components: filecache Affects Versions: 0.20.1 Environment: CentOS Reporter: Marc Colosimo This is a continuation of http://issues.apache.org/jira/browse/HADOOP-5635 (JIRA wouldn't let me edit that one). I found another issue with DistributedCache using something besides HDFS. In my case I have TWO active file systems, with HDFS being the default file system. My fix includes two additional changes (from HADOOP-5635) to get it to work with another filesystem scheme (plus the changes from the original patch). I've tested this an it works with my code on HDFS with another filesystem. I have similar changes to mapreduce.filecacheTaskDistributedCacheManager and TrackerDistributedCacheManager (0.22.0). Basically, URI.getPath() is called instead of URI.toString(). toString returns the scheme plus path which is important in finding the file to copy (getting the file system). Otherwise it searches the default file system (in this case HDFS) for the file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Private, LimitedPrivate and contrib modules
On Nov 5, 2009, at 12:15 PM, Dhruba Borthakur wrote: Hi sanjay, Most of the contrib modules are in the same package as their containers. For example, the fair share scheduler is in contrib but its package name is org.apache.hadoop.mapred. Doesn't this mean that the fair-share scheduler code can use LimitedPrivate methods from org.apache.hadoop.mapred package? The package/java public private does not imply an API's visibility. We have often made certain API's java-public because of java's limitations on visibility rules (eg. if java had subpackage-private, then a number of apis would change from java- public to java-subpackage-private. Generally contrib should be using API that are audience-public. The FS/C schedulers are very good examples. I assume that it is using the scheduler's plugin interfaces. Is it using any other internal interfaces of MR? Plugin interfaces or Abstract classes that are to be used by implementors are current planned to be marked audience-public. (See AbstractFileSystem). Makes sense? As a side note: Perhaps we need to add tag to the audience-public of applicationUse implementorsOfInteface or something like that. (like the tag for limited-private). sanjay thanks, dhruba On Thu, Nov 5, 2009 at 11:05 AM, Sanjay Radia sra...@yahoo-inc.com wrote: Sorry for the late reply .. I missed in in my inbox. On Sep 18, 2009, at 1:29 PM, Tom White wrote: I'm trying to better understand the meaning of the annotations defined in org.apache.hadoop.classification.InterfaceAudience. 1. Private is documented as being Intended for use only within Hadoop itself. Does this mean the whole Hadoop project, or the subproject that the annotated element is in? (Or another scope?) Private means private to the subproject (ie private to HDFS, to MR, etc). We should update the doc to clarify this. 2. Is a contrib module considered to be a part of the subproject? For example, if something is marked as LimitedPrivate in Common say, with intended audience MapReduce, can MapReduce contrib modules use it? I believe the answer should be no. Or, assuming the second meaning for 1, if something is marked as Private in MapReduce, can MapReduce contrib modules use it? Again I believe the answer should be no. Any concrete examples that we can use to examine/discuss this further Thanks, Tom -- Connect to me at http://www.facebook.com/dhruba
[jira] Created: (HADOOP-6366) Reduce ivy console output to ovservable level
Reduce ivy console output to ovservable level - Key: HADOOP-6366 URL: https://issues.apache.org/jira/browse/HADOOP-6366 Project: Hadoop Common Issue Type: Improvement Reporter: Konstantin Boudnik Fix For: 0.22.0 It is very hard to see what's going in the build because ivy is literally flood the console with nonsensical messages... -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
