[jira] [Commented] (HIVE-4739) Create public parallel test environment
[ https://issues.apache.org/jira/browse/HIVE-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714360#comment-13714360 ] Carl Steinbach commented on HIVE-4739: -- [~brocknoland] Ditto. Thanks! Create public parallel test environment --- Key: HIVE-4739 URL: https://issues.apache.org/jira/browse/HIVE-4739 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland See discussion here http://mail-archives.apache.org/mod_mbox/hive-dev/201306.mbox/%3CCAENxBwx%3Dc4VFLhR1zXZ3uWbP2BUFZc%2BZuiqF4SZY5zUZ9%2BUcxQ%40mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4739) Create public parallel test environment
[ https://issues.apache.org/jira/browse/HIVE-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13713909#comment-13713909 ] Brock Noland commented on HIVE-4739: [~ashutoshc] [~cwsteinbach] or any other committer did you guys want admin accounts on this testing infrastructure? If so just shoot me an email with the username you'd like and I'll create an account and password for you. Create public parallel test environment --- Key: HIVE-4739 URL: https://issues.apache.org/jira/browse/HIVE-4739 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland See discussion here http://mail-archives.apache.org/mod_mbox/hive-dev/201306.mbox/%3CCAENxBwx%3Dc4VFLhR1zXZ3uWbP2BUFZc%2BZuiqF4SZY5zUZ9%2BUcxQ%40mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4739) Create public parallel test environment
[ https://issues.apache.org/jira/browse/HIVE-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13713915#comment-13713915 ] Ashutosh Chauhan commented on HIVE-4739: Thanks [~brocknoland] for setting this up. I will shoot you an email. Create public parallel test environment --- Key: HIVE-4739 URL: https://issues.apache.org/jira/browse/HIVE-4739 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland See discussion here http://mail-archives.apache.org/mod_mbox/hive-dev/201306.mbox/%3CCAENxBwx%3Dc4VFLhR1zXZ3uWbP2BUFZc%2BZuiqF4SZY5zUZ9%2BUcxQ%40mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4739) Create public parallel test environment
[ https://issues.apache.org/jira/browse/HIVE-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13711217#comment-13711217 ] Brock Noland commented on HIVE-4739: Hey guys, This is pretty much ready to go. Any committers that would like admin access to this infrastructure should send me there uid and public ssh key. This can be shared by encrypting a message (http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0x9DC27AFB8E991CC5) with pgp or sending me an email with a service such as https://www.sendinc.com/. Brock Create public parallel test environment --- Key: HIVE-4739 URL: https://issues.apache.org/jira/browse/HIVE-4739 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland See discussion here http://mail-archives.apache.org/mod_mbox/hive-dev/201306.mbox/%3CCAENxBwx%3Dc4VFLhR1zXZ3uWbP2BUFZc%2BZuiqF4SZY5zUZ9%2BUcxQ%40mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4739) Create public parallel test environment
[ https://issues.apache.org/jira/browse/HIVE-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13711232#comment-13711232 ] Brock Noland commented on HIVE-4739: uid should be username above. Create public parallel test environment --- Key: HIVE-4739 URL: https://issues.apache.org/jira/browse/HIVE-4739 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland See discussion here http://mail-archives.apache.org/mod_mbox/hive-dev/201306.mbox/%3CCAENxBwx%3Dc4VFLhR1zXZ3uWbP2BUFZc%2BZuiqF4SZY5zUZ9%2BUcxQ%40mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4739) Create public parallel test environment
[ https://issues.apache.org/jira/browse/HIVE-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13701639#comment-13701639 ] Brock Noland commented on HIVE-4739: [~appodictic] OK I think this is ready. To summarize * A dedicated ec2 medium instance runs a webservice which is used to kick of the build * The build starts up 8 ec2 spot instances and then runs the build (these stick around for a half hour waiting for another build) * It seems to work well, see more details here: https://issues.apache.org/jira/browse/HIVE-4675?focusedCommentId=13701636page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13701636 * This is extremely cheap and handling multiple ec2 keys would be complex. Therefore let's just charge this to my employer for now and then later we can add multiple ec2 key support to ptest2. * Once HIVE-4815 is resolved I can create the automated patch available testing I am ready to create accounts for hive committers that are interested. In order to protect the ec2 key, the account will not have full sudo access. However, the account will be able to: * Stop/Start/Restart the ptest2 service * Push the latest changes to ptest2 from trunk * Look at all logs How does this sound? Create public parallel test environment --- Key: HIVE-4739 URL: https://issues.apache.org/jira/browse/HIVE-4739 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland See discussion here http://mail-archives.apache.org/mod_mbox/hive-dev/201306.mbox/%3CCAENxBwx%3Dc4VFLhR1zXZ3uWbP2BUFZc%2BZuiqF4SZY5zUZ9%2BUcxQ%40mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4739) Create public parallel test environment
[ https://issues.apache.org/jira/browse/HIVE-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13701645#comment-13701645 ] Brock Noland commented on HIVE-4739: Also in case it wasn't clear above, my plan would be to create individual accounts for hive committers interested in working with this infrastructure. Create public parallel test environment --- Key: HIVE-4739 URL: https://issues.apache.org/jira/browse/HIVE-4739 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland See discussion here http://mail-archives.apache.org/mod_mbox/hive-dev/201306.mbox/%3CCAENxBwx%3Dc4VFLhR1zXZ3uWbP2BUFZc%2BZuiqF4SZY5zUZ9%2BUcxQ%40mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4739) Create public parallel test environment
[ https://issues.apache.org/jira/browse/HIVE-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13692992#comment-13692992 ] Brock Noland commented on HIVE-4739: I hope to use Jenkins parameterized builds in terms of the end user interface. Basically you'd go the Apache-Hive-Trunk-Patch-Build and then paste in the URL of the patch or upload the patch file. In order to implement this I think we'd need some kind of service simply to keep the EC2 API key semi-private. That is putting it on the Jenkins build slaves probably isn't the best idea. Additionally if we were going to allow contributors to test patches this way a service might be required... Brock Create public parallel test environment --- Key: HIVE-4739 URL: https://issues.apache.org/jira/browse/HIVE-4739 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland See discussion here http://mail-archives.apache.org/mod_mbox/hive-dev/201306.mbox/%3CCAENxBwx%3Dc4VFLhR1zXZ3uWbP2BUFZc%2BZuiqF4SZY5zUZ9%2BUcxQ%40mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4739) Create public parallel test environment
[ https://issues.apache.org/jira/browse/HIVE-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13693614#comment-13693614 ] Edward Capriolo commented on HIVE-4739: --- I also like the way the Hadoop does it. As an issue is marked 'patch available' it kicks in. Then it reply's back the jira with the test result. Not sure if that is done by an external system querying jira or some type of jira hook. Create public parallel test environment --- Key: HIVE-4739 URL: https://issues.apache.org/jira/browse/HIVE-4739 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland See discussion here http://mail-archives.apache.org/mod_mbox/hive-dev/201306.mbox/%3CCAENxBwx%3Dc4VFLhR1zXZ3uWbP2BUFZc%2BZuiqF4SZY5zUZ9%2BUcxQ%40mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4739) Create public parallel test environment
[ https://issues.apache.org/jira/browse/HIVE-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13691189#comment-13691189 ] Edward Capriolo commented on HIVE-4739: --- Is the concept that we can simply send an SVN URL, and a patch URL to a service and it runs svn co ; ant test ; ant testreport? Create public parallel test environment --- Key: HIVE-4739 URL: https://issues.apache.org/jira/browse/HIVE-4739 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland See discussion here http://mail-archives.apache.org/mod_mbox/hive-dev/201306.mbox/%3CCAENxBwx%3Dc4VFLhR1zXZ3uWbP2BUFZc%2BZuiqF4SZY5zUZ9%2BUcxQ%40mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4739) Create public parallel test environment
[ https://issues.apache.org/jira/browse/HIVE-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13685670#comment-13685670 ] Brock Noland commented on HIVE-4739: My employer, Cloudera, has agreed to sponsor a public/open test cluster for the Hive project. The current thought is to run the ptest framework using Apache JClouds (incubating) to spin up spot instances via Jenkins for each run. I will be working through setting this up. Just to be clear, once setup it will be available for Hive community use. If anyone else is interested in helping to sponsor this project, just comment here and we can work out the details. Create public parallel test environment --- Key: HIVE-4739 URL: https://issues.apache.org/jira/browse/HIVE-4739 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland See discussion here http://mail-archives.apache.org/mod_mbox/hive-dev/201306.mbox/%3CCAENxBwx%3Dc4VFLhR1zXZ3uWbP2BUFZc%2BZuiqF4SZY5zUZ9%2BUcxQ%40mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4739) Create public parallel test environment
[ https://issues.apache.org/jira/browse/HIVE-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13685989#comment-13685989 ] Edward Capriolo commented on HIVE-4739: --- +1. My dream is to only have to do code reviews, that testing patches would be completely automated. It will take a lot of burden off the committers and make the project more fair and a lower barrier for entry for newer people. I am interested to help sponsor. There is an important point here. By OPEN the platform has to be OPEN. It can not be open like fabricator/arc, when that process is broken there is no access and no path for support. It needs to be a service running somewhere that any hive committer can access and administer. Create public parallel test environment --- Key: HIVE-4739 URL: https://issues.apache.org/jira/browse/HIVE-4739 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland See discussion here http://mail-archives.apache.org/mod_mbox/hive-dev/201306.mbox/%3CCAENxBwx%3Dc4VFLhR1zXZ3uWbP2BUFZc%2BZuiqF4SZY5zUZ9%2BUcxQ%40mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4739) Create public parallel test environment
[ https://issues.apache.org/jira/browse/HIVE-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13686062#comment-13686062 ] Brock Noland commented on HIVE-4739: Hi, We have the same dream! :) Regarding sponsorship, I am working on a POC with spot instances. If spot instances work there are two items of note: 1) Sponsorship will be more difficult since they'll be created on demand. I think we'd have to managed multiple EC2 Api keys. 2) It looks like it will be very, very cheap. Regarding OPEN - I agree. If the instances are created on-demand, the only item I think we'll need to figure out is how to store the EC2 API key. I am not too versed on EC2 API keys but I am hoping we can create a key for this specific application. Brock Create public parallel test environment --- Key: HIVE-4739 URL: https://issues.apache.org/jira/browse/HIVE-4739 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland See discussion here http://mail-archives.apache.org/mod_mbox/hive-dev/201306.mbox/%3CCAENxBwx%3Dc4VFLhR1zXZ3uWbP2BUFZc%2BZuiqF4SZY5zUZ9%2BUcxQ%40mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4739) Create public parallel test environment
[ https://issues.apache.org/jira/browse/HIVE-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13684820#comment-13684820 ] Brock Noland commented on HIVE-4739: As I see it there are three phases: 1) Executing the post-commit tests on the parallel infrastructure 2) Providing a manual patch test build 3) Implementing automated patch testing I am interesting this work. Create public parallel test environment --- Key: HIVE-4739 URL: https://issues.apache.org/jira/browse/HIVE-4739 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland See discussion here http://mail-archives.apache.org/mod_mbox/hive-dev/201306.mbox/%3CCAENxBwx%3Dc4VFLhR1zXZ3uWbP2BUFZc%2BZuiqF4SZY5zUZ9%2BUcxQ%40mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
