[jira] [Commented] (HDFS-4262) Backport HTTPFS to Branch 1
[ https://issues.apache.org/jira/browse/HDFS-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083637#comment-16083637 ] Andras Bokor commented on HDFS-4262: Seems like obsolete. Is it still intended to fix? > Backport HTTPFS to Branch 1 > --- > > Key: HDFS-4262 > URL: https://issues.apache.org/jira/browse/HDFS-4262 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode > Environment: IBM JDK, RHEL 6.3 >Reporter: Eric Yang >Assignee: Yu Li > Attachments: 01-retrofit-httpfs-cdh3u4-for-hadoop1.patch, > 02-cookie-from-authenticated-url-is-not-getting-to-auth-filter.patch, > 03-resolve-proxyuser-related-issue.patch, HDFS-4262-github.patch > > > There are interests to backport HTTPFS back to Hadoop 1 branch. After the > initial investigation, there're quite some changes in HDFS-2178, and several > related patches, including: > HDFS-2284 Write Http access to HDFS > HDFS-2646 Hadoop HttpFS introduced 4 findbug warnings > HDFS-2649 eclipse:eclipse build fails for hadoop-hdfs-httpfs > HDFS-2657 TestHttpFSServer and TestServerWebApp are failing on trunk > HDFS-2658 HttpFS introduced 70 javadoc warnings > The most challenge of backporting is all these patches, including HDFS-2178 > are for 2.X, which code base has been refactored a lot and quite different > from 1.X, so it seems we have to backport the changes manually. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-4262) Backport HTTPFS to Branch 1
[ https://issues.apache.org/jira/browse/HDFS-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13564039#comment-13564039 ] Matt Foley commented on HDFS-4262: -- Since this is still underway, please target 1.2 (branch-1) rather than 1.1. Thanks. Backport HTTPFS to Branch 1 --- Key: HDFS-4262 URL: https://issues.apache.org/jira/browse/HDFS-4262 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Environment: IBM JDK, RHEL 6.3 Reporter: Eric Yang Assignee: Yu Li Attachments: 01-retrofit-httpfs-cdh3u4-for-hadoop1.patch, 02-cookie-from-authenticated-url-is-not-getting-to-auth-filter.patch, 03-resolve-proxyuser-related-issue.patch, HDFS-4262-github.patch There are interests to backport HTTPFS back to Hadoop 1 branch. After the initial investigation, there're quite some changes in HDFS-2178, and several related patches, including: HDFS-2284 Write Http access to HDFS HDFS-2646 Hadoop HttpFS introduced 4 findbug warnings HDFS-2649 eclipse:eclipse build fails for hadoop-hdfs-httpfs HDFS-2657 TestHttpFSServer and TestServerWebApp are failing on trunk HDFS-2658 HttpFS introduced 70 javadoc warnings The most challenge of backporting is all these patches, including HDFS-2178 are for 2.X, which code base has been refactored a lot and quite different from 1.X, so it seems we have to backport the changes manually. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4262) Backport HTTPFS to Branch 1
[ https://issues.apache.org/jira/browse/HDFS-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13510680#comment-13510680 ] Yu Li commented on HDFS-4262: - Thanks Alejandro. After applying the two patches you supplied, UT result in my env is like: === Tests in error: testOperation[0](org.apache.hadoop.fs.http.client.TestWebhdfsFileSystem): java.io.IOException: Server returned HTTP response code: 500 for URL: http://bdvm072.svl.ibm.com:48763/webhdfs/v1/?op=GETHOMEDIRECTORYdoas=biadmin testOperationDoAs[0](org.apache.hadoop.fs.http.client.TestWebhdfsFileSystem): java.io.IOException: Server returned HTTP response code: 401 for URL: http://bdvm072.svl.ibm.com:57519/webhdfs/v1/?op=GETHOMEDIRECTORYdoas=user1 testOperation[0](org.apache.hadoop.fs.http.client.TestHttpFSFileSystem): java.io.IOException: Server returned HTTP response code: 500 for URL: http://bdvm072.svl.ibm.com:57757/webhdfs/v1/?op=GETHOMEDIRECTORYdoas=biadmin testOperationDoAs[0](org.apache.hadoop.fs.http.client.TestHttpFSFileSystem): java.io.IOException: Server returned HTTP response code: 401 for URL: http://bdvm072.svl.ibm.com:56289/webhdfs/v1/?op=GETHOMEDIRECTORYdoas=user1 Tests run: 177, Failures: 0, Errors: 4, Skipped: 0 === Then I did some investigation and made another patch, 03-resolve-proxyuser-related-issue.patch, which could resolve the UT failure. I also merged the three patches together to HDFS-4262-github.patch, as attached. After getting all UT pass, I also build-out a httpfs tar ball and tested it on a hadoop-1.0.3 environment, and most function worked. However, I found the API documents, either the ones attached with HDFS-2178 or on http://cloudera.github.com/httpfs/UsingHttpTools.html, are out of date. Like to get the name dir of one specified user, the request should be: picurl -X GET http://shihc024-public.cn.ibm.com:14000/webhdfs/v1?user.name=biadminop=gethomedirectory;/i/p rather than picurl -i http://HTTPFSHOSTNAME:14000?user.name=babuop=homedir/i/p Anybody could tell me where I can get the latest API doc, so I could apply a full sanity test? Backport HTTPFS to Branch 1 --- Key: HDFS-4262 URL: https://issues.apache.org/jira/browse/HDFS-4262 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Environment: IBM JDK, RHEL 6.3 Reporter: Eric Yang Assignee: Yu Li Attachments: 01-retrofit-httpfs-cdh3u4-for-hadoop1.patch, 02-cookie-from-authenticated-url-is-not-getting-to-auth-filter.patch, 03-resolve-proxyuser-related-issue.patch, HDFS-4262-github.patch There are interests to backport HTTPFS back to Hadoop 1 branch. After the initial investigation, there're quite some changes in HDFS-2178, and several related patches, including: HDFS-2284 Write Http access to HDFS HDFS-2646 Hadoop HttpFS introduced 4 findbug warnings HDFS-2649 eclipse:eclipse build fails for hadoop-hdfs-httpfs HDFS-2657 TestHttpFSServer and TestServerWebApp are failing on trunk HDFS-2658 HttpFS introduced 70 javadoc warnings The most challenge of backporting is all these patches, including HDFS-2178 are for 2.X, which code base has been refactored a lot and quite different from 1.X, so it seems we have to backport the changes manually. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4262) Backport HTTPFS to Branch 1
[ https://issues.apache.org/jira/browse/HDFS-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13510709#comment-13510709 ] Yu Li commented on HDFS-4262: - Next step I will try to put the httpfs source codes into src/contrib, and change to use ant(build.xml) instead of maven(pom.xml) to build. Any comments, please let me know, thanks! Backport HTTPFS to Branch 1 --- Key: HDFS-4262 URL: https://issues.apache.org/jira/browse/HDFS-4262 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Environment: IBM JDK, RHEL 6.3 Reporter: Eric Yang Assignee: Yu Li Attachments: 01-retrofit-httpfs-cdh3u4-for-hadoop1.patch, 02-cookie-from-authenticated-url-is-not-getting-to-auth-filter.patch, 03-resolve-proxyuser-related-issue.patch, HDFS-4262-github.patch There are interests to backport HTTPFS back to Hadoop 1 branch. After the initial investigation, there're quite some changes in HDFS-2178, and several related patches, including: HDFS-2284 Write Http access to HDFS HDFS-2646 Hadoop HttpFS introduced 4 findbug warnings HDFS-2649 eclipse:eclipse build fails for hadoop-hdfs-httpfs HDFS-2657 TestHttpFSServer and TestServerWebApp are failing on trunk HDFS-2658 HttpFS introduced 70 javadoc warnings The most challenge of backporting is all these patches, including HDFS-2178 are for 2.X, which code base has been refactored a lot and quite different from 1.X, so it seems we have to backport the changes manually. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4262) Backport HTTPFS to Branch 1
[ https://issues.apache.org/jira/browse/HDFS-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13510726#comment-13510726 ] Alejandro Abdelnur commented on HDFS-4262: -- Yu Li, The REST API is at http://hadoop.apache.org/docs/r1.0.4/webhdfs.html Regarding the change you made in the FileAccessService, I think is it incorrect, even if the HTTPFS client is the same user as the HTTP server, it should create a proxy user. This works fine in Hadoop 2, so it should here. The changes of the proxy user configuration in the test are OK if the version of Hadoop 1 you are backporting to supports wildcard '*', I've done those changes because the last version of Hadoop 1 I've tried the backport did not support it. Regarding changing the build to use Ant, it is your call. Personally, I'd would just leave it as Maven and call Maven from Ant for this contrib. Regardless, I don't know what is your idea on how this will layer on the Hadoop 1 layout, but you may have to change the location of dirs. What I would do is to have a subdir httpfs in the root of the Hadoop 1 distribution and under it have the current layout. This would mean minimal changes to the Hadoop 2 HTTPFS, meaning that all backports will be easier to resolve (in case you need to). Thx. Backport HTTPFS to Branch 1 --- Key: HDFS-4262 URL: https://issues.apache.org/jira/browse/HDFS-4262 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Environment: IBM JDK, RHEL 6.3 Reporter: Eric Yang Assignee: Yu Li Attachments: 01-retrofit-httpfs-cdh3u4-for-hadoop1.patch, 02-cookie-from-authenticated-url-is-not-getting-to-auth-filter.patch, 03-resolve-proxyuser-related-issue.patch, HDFS-4262-github.patch There are interests to backport HTTPFS back to Hadoop 1 branch. After the initial investigation, there're quite some changes in HDFS-2178, and several related patches, including: HDFS-2284 Write Http access to HDFS HDFS-2646 Hadoop HttpFS introduced 4 findbug warnings HDFS-2649 eclipse:eclipse build fails for hadoop-hdfs-httpfs HDFS-2657 TestHttpFSServer and TestServerWebApp are failing on trunk HDFS-2658 HttpFS introduced 70 javadoc warnings The most challenge of backporting is all these patches, including HDFS-2178 are for 2.X, which code base has been refactored a lot and quite different from 1.X, so it seems we have to backport the changes manually. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4262) Backport HTTPFS to Branch 1
[ https://issues.apache.org/jira/browse/HDFS-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509798#comment-13509798 ] Alejandro Abdelnur commented on HDFS-4262: -- You may want to look at https://github.com/cloudera/httpfs, that is a backport to cdh3 Hadoop I've done a while ago. At the time I've tried to get things working with Hadoop 1 but run into some test issues with MiniDFS and I did not have time to follow up. If you are interested, I can upload the delta patch on top of the git repo with the work I've done for Hadoop 1 branch. Backport HTTPFS to Branch 1 --- Key: HDFS-4262 URL: https://issues.apache.org/jira/browse/HDFS-4262 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Environment: IBM JDK, RHEL 6.3 Reporter: Eric Yang Assignee: Yu Li There are interests to backport HTTPFS back to Hadoop 1 branch. After the initial investigation, there're quite some changes in HDFS-2178, and several related patches, including: HDFS-2284 Write Http access to HDFS HDFS-2646 Hadoop HttpFS introduced 4 findbug warnings HDFS-2649 eclipse:eclipse build fails for hadoop-hdfs-httpfs HDFS-2657 TestHttpFSServer and TestServerWebApp are failing on trunk HDFS-2658 HttpFS introduced 70 javadoc warnings The most challenge of backporting is all these patches, including HDFS-2178 are for 2.X, which code base has been refactored a lot and quite different from 1.X, so it seems we have to backport the changes manually. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4262) Backport HTTPFS to Branch 1
[ https://issues.apache.org/jira/browse/HDFS-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509804#comment-13509804 ] Yu Li commented on HDFS-4262: - Yes, Alejandro, please upload your delta patch, I believe it's a good base to work upon, thx! 在 2012/12/4 11:29 PM，Alejandro Abdelnur (JIRA) j...@apache.org写道： Backport HTTPFS to Branch 1 --- Key: HDFS-4262 URL: https://issues.apache.org/jira/browse/HDFS-4262 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Environment: IBM JDK, RHEL 6.3 Reporter: Eric Yang Assignee: Yu Li There are interests to backport HTTPFS back to Hadoop 1 branch. After the initial investigation, there're quite some changes in HDFS-2178, and several related patches, including: HDFS-2284 Write Http access to HDFS HDFS-2646 Hadoop HttpFS introduced 4 findbug warnings HDFS-2649 eclipse:eclipse build fails for hadoop-hdfs-httpfs HDFS-2657 TestHttpFSServer and TestServerWebApp are failing on trunk HDFS-2658 HttpFS introduced 70 javadoc warnings The most challenge of backporting is all these patches, including HDFS-2178 are for 2.X, which code base has been refactored a lot and quite different from 1.X, so it seems we have to backport the changes manually. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4262) Backport HTTPFS to Branch 1
[ https://issues.apache.org/jira/browse/HDFS-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509503#comment-13509503 ] Suresh Srinivas commented on HDFS-4262: --- +1. This feature is useful along with WebHDFS that already is available in branch-1. Backport HTTPFS to Branch 1 --- Key: HDFS-4262 URL: https://issues.apache.org/jira/browse/HDFS-4262 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Environment: IBM JDK, RHEL 6.3 Reporter: Eric Yang Assignee: Yu Li There are interests to backport HTTPFS back to Hadoop 1 branch. After the initial investigation, there're quite some changes in HDFS-2178, and several related patches, including: HDFS-2284 Write Http access to HDFS HDFS-2646 Hadoop HttpFS introduced 4 findbug warnings HDFS-2649 eclipse:eclipse build fails for hadoop-hdfs-httpfs HDFS-2657 TestHttpFSServer and TestServerWebApp are failing on trunk HDFS-2658 HttpFS introduced 70 javadoc warnings The most challenge of backporting is all these patches, including HDFS-2178 are for 2.X, which code base has been refactored a lot and quite different from 1.X, so it seems we have to backport the changes manually. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
