[jira] [Resolved] (HADOOP-8885) Need to add fs shim to use QFS

2012-10-26 Thread thilee (JIRA)
/ Instructions for obtaining and using the QFS Hadoop plugin are at https://github.com/quantcast/qfs/wiki/Migration-Guide Need to add fs shim to use QFS -- Key: HADOOP-8885 URL: https://issues.apache.org/jira/browse/HADOOP-8885

Re: Need to add fs shim to use QFS

2012-10-26 Thread Steve Loughran
On 26 October 2012 01:24, Thilee Subramaniam thi...@quantcast.com wrote: We have made the changes recommended here, and made available a 'Hadoop QFS jar' with QFS. This plugin and the QFS libraries will be maintained released by the QFS open-source project. Please see the download and

Re: Need to add fs shim to use QFS

2012-10-25 Thread Thilee Subramaniam
On 10/11/12 8:01 AM, Steve Loughran ste...@hortonworks.com wrote: On 11 October 2012 00:34, Thilee Subramaniam thi...@quantcast.com wrote: My initial goal was to make Hadoop use QFS the same way it used KFS. Since Hadoop branch-1 had lib/kfs.xx.jar, I was expecting to include a qfs.x.x.jar

Re: Need to add fs shim to use QFS

2012-10-11 Thread Steve Loughran
Good point Steve. This touches on the larger issue of whether it makes sense to host FS clients for other file systems in Hadoop itself. I agree with what I think you're getting which is - if we can handle the testing and integration via external dependencies it would probably be better to

Re: Need to add fs shim to use QFS

2012-10-11 Thread Steve Loughran
On 11 October 2012 00:34, Thilee Subramaniam thi...@quantcast.com wrote: My initial goal was to make Hadoop use QFS the same way it used KFS. Since Hadoop branch-1 had lib/kfs.xx.jar, I was expecting to include a qfs.x.x.jar in the Hadoop release; my first patch was to use such jar. But now

Re: Need to add fs shim to use QFS

2012-10-10 Thread Harsh J
Hi Steve, Check out https://issues.apache.org/jira/browse/HADOOP-8886 for the KFS removal. Seems relevant to your question here. On Wed, Oct 10, 2012 at 12:52 AM, Steve Loughran ste...@hortonworks.com wrote: On 5 October 2012 18:27, Thilee Subramaniam thi...@quantcast.com wrote: We at

Re: Need to add fs shim to use QFS

2012-10-10 Thread Thilee Subramaniam
Hi Steve, Like Harsh said, HADOOP-8886 addresses removing KFS from apache tree. But I interpret your suggestion as 'moving qfs.jar out of apache tree, and keeping the jar in possibly a maven repo externally. The new fs shim for QFS will pull this jar from the repo upon compilation etc.'. Please

Re: Need to add fs shim to use QFS

2012-10-10 Thread Steve Loughran
On 10 October 2012 16:03, Thilee Subramaniam thi...@quantcast.com wrote: Hi Steve, Like Harsh said, HADOOP-8886 addresses removing KFS from apache tree. But I interpret your suggestion as 'moving qfs.jar out of apache tree, and keeping the jar in possibly a maven repo externally. The new fs

Re: Need to add fs shim to use QFS

2012-10-10 Thread Thilee Subramaniam
On 10/10/12 12:14 PM, Steve Loughran ste...@hortonworks.com wrote: On 10 October 2012 16:03, Thilee Subramaniam thi...@quantcast.com wrote: Hi Steve, Like Harsh said, HADOOP-8886 addresses removing KFS from apache tree. But I interpret your suggestion as 'moving qfs.jar out of apache tree,

Re: Need to add fs shim to use QFS

2012-10-09 Thread Thilee Subramaniam
Hi Eli, I have attached two patches and submitted them as well. Will the integration engine automatically process 'branch-1' patch or should I do anything spcific for released versions? I set the following for the Fix versions field upon patch submission: 1.0.2, 1.0.3, 0.23.3, 2.0.0-alpha,

Re: Need to add fs shim to use QFS

2012-10-09 Thread Steve Loughran
On 5 October 2012 18:27, Thilee Subramaniam thi...@quantcast.com wrote: We at Quantcast have released QFS 1.0 (Quantcast File System) to open source. This is based on the KFS 0.5 (Kosmos Distributed File System), a C++ distributed filesystem implementation. KFS plugs into Apache Hadoop via

[jira] [Created] (HADOOP-8885) Need to add fs shim to use QFS

2012-10-05 Thread thilee (JIRA)
thilee created HADOOP-8885: -- Summary: Need to add fs shim to use QFS Key: HADOOP-8885 URL: https://issues.apache.org/jira/browse/HADOOP-8885 Project: Hadoop Common Issue Type: New Feature

Need to add fs shim to use QFS

2012-10-05 Thread Thilee Subramaniam
We at Quantcast have released QFS 1.0 (Quantcast File System) to open source. This is based on the KFS 0.5 (Kosmos Distributed File System), a C++ distributed filesystem implementation. KFS plugs into Apache Hadoop via the 'kfs' shim that is part of Hadoop codebase. QFS has added support for

Re: Need to add fs shim to use QFS

2012-10-05 Thread Eli Collins
Hey Thilee, Thanks for contributing. We don't process pull request on the git mirrors, please upload a patch against trunk and branch-1 if you'd like this included in Hadoop 1.x and 2.x releases. More info here: http://wiki.apache.org/hadoop/HowToContribute Thanks, Eli On Fri, Oct 5, 2012 at

Re: Need to add fs shim to use QFS

2012-10-05 Thread Thilee Subramaniam
Thank you Eli. I have just uploaded the patch for branch-1 to JIRA. I'll do the same for trunk soon. If you have any comments or questions on the patch please let me know. Thanks, - Thilee On 10/5/12 10:32 AM, Eli Collins e...@cloudera.com wrote: Hey Thilee, Thanks for contributing. We don't