[jira] [Commented] (HIVE-15041) Specify GCE network name on Hive ptest
[ https://issues.apache.org/jira/browse/HIVE-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626098#comment-15626098 ] Siddharth Seth commented on HIVE-15041: --- Thoughts on temporarily re-creating the ContextProvider to see if that helps?, while this issue is debugged. Commented on HIVE-15102. > Specify GCE network name on Hive ptest > -- > > Key: HIVE-15041 > URL: https://issues.apache.org/jira/browse/HIVE-15041 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-15041.1.patch, HIVE-15041.2.patch > > > NO PRECOMMIT TESTS > A new option on cloudhost.properties should be added to specify the GCE > network name: > # GCE network option > network = > https://www.googleapis.com/compute/v1/projects//global/networks/default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15041) Specify GCE network name on Hive ptest
[ https://issues.apache.org/jira/browse/HIVE-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15625842#comment-15625842 ] Sergio Peña commented on HIVE-15041: When testing GCE with jclouds, I found out that 'jclouds-hive-ptest-slaves' is created automatically by jclouds all the time. I saw that the network parameter I added is not working as expected, and jclouds is still choosing its own network. Not sure what happened here. Need to dig more on this networking issue. In the meantime, if you can review HIVE-15102 for the resued IP issue. I created a test case to replicate that. > Specify GCE network name on Hive ptest > -- > > Key: HIVE-15041 > URL: https://issues.apache.org/jira/browse/HIVE-15041 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-15041.1.patch, HIVE-15041.2.patch > > > NO PRECOMMIT TESTS > A new option on cloudhost.properties should be added to specify the GCE > network name: > # GCE network option > network = > https://www.googleapis.com/compute/v1/projects//global/networks/default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15041) Specify GCE network name on Hive ptest
[ https://issues.apache.org/jira/browse/HIVE-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623687#comment-15623687 ] Siddharth Seth commented on HIVE-15041: --- Got it. Was looking at the error. A post call to projects/gcp-hive-upstream/global/firewalls complaining about gcp-hive-upstream/global/networks/jclouds-hive-ptest-slaves not being available. (Should this have been gcp-hive-upstream/global/networks/hive-ptest-slaves). Not sure where the jclouds gets inserted. Anyway - seems like any refresh causes this problem. Till this figured out, one possibility would be to try and re-created the ExecutionContextProvider once the timeout has expired, instead of attempting to re-use the same executionContextProvider to get more nodes. Couldn't trace down what changes in between which could cause this to fail. The config etc could change a bit to make parameters final for thread visibility - but I don't think that is the problem. > Specify GCE network name on Hive ptest > -- > > Key: HIVE-15041 > URL: https://issues.apache.org/jira/browse/HIVE-15041 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-15041.1.patch, HIVE-15041.2.patch > > > NO PRECOMMIT TESTS > A new option on cloudhost.properties should be added to specify the GCE > network name: > # GCE network option > network = > https://www.googleapis.com/compute/v1/projects//global/networks/default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15041) Specify GCE network name on Hive ptest
[ https://issues.apache.org/jira/browse/HIVE-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623608#comment-15623608 ] Sergio Peña commented on HIVE-15041: [~sseth] Seems that running a 'git pull origin' will apply all latest commits clean if the changes on the code do not conflict. Regarding stricthostkeychecking, the /etc/ssh/ssh_config is configured with the following lines to avoid errors with SSH: StrictHostKeyChecking no UserKnownHostsFile=/dev/nul > Specify GCE network name on Hive ptest > -- > > Key: HIVE-15041 > URL: https://issues.apache.org/jira/browse/HIVE-15041 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-15041.1.patch, HIVE-15041.2.patch > > > NO PRECOMMIT TESTS > A new option on cloudhost.properties should be added to specify the GCE > network name: > # GCE network option > network = > https://www.googleapis.com/compute/v1/projects//global/networks/default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15041) Specify GCE network name on Hive ptest
[ https://issues.apache.org/jira/browse/HIVE-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623465#comment-15623465 ] Siddharth Seth commented on HIVE-15041: --- Didn't know the patch was already running. Wouldn't it reset each time there's an update and restart if it's not committed? I'll try taking a look as well. In terms of the IP problem - would making use of hostnames instead help? If GCE is giving the same IPs - I'm surprised we haven't run into problems with entries in .ssh/known_hosts. (Some part of the code works with stricthostkeychecking set to no, other parts don't) > Specify GCE network name on Hive ptest > -- > > Key: HIVE-15041 > URL: https://issues.apache.org/jira/browse/HIVE-15041 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-15041.1.patch > > > NO PRECOMMIT TESTS > A new option on cloudhost.properties should be added to specify the GCE > network name: > # GCE network option > network = > https://www.googleapis.com/compute/v1/projects//global/networks/default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15041) Specify GCE network name on Hive ptest
[ https://issues.apache.org/jira/browse/HIVE-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15623096#comment-15623096 ] Sergio Peña commented on HIVE-15041: The patch has been running on ptest since I created this jira, but I don't see any difference on fixing the issues. There was one thing I added to the patch that I haven't updated, but it will not add any improvement. I will include the line, and commit it just to keep it in the branch for future issues with the network. Another thing while debugging is that seems there are issues with IPs already used before. Ptest saves a list of Hosts IPs that are destroyed when they are not used (normal operation), but then for the next batch of Hosts created, GCE is giving some IPs that we already saved in a terminated list. So far is good, but the problem comes when a ptest background check runs (every 1h) that looks into those "already" terminated hosts, and it destroys them again. I'll dig more into this issue and create a jira. I haven't had too much time to work on this though. > Specify GCE network name on Hive ptest > -- > > Key: HIVE-15041 > URL: https://issues.apache.org/jira/browse/HIVE-15041 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-15041.1.patch > > > NO PRECOMMIT TESTS > A new option on cloudhost.properties should be added to specify the GCE > network name: > # GCE network option > network = > https://www.googleapis.com/compute/v1/projects//global/networks/default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15041) Specify GCE network name on Hive ptest
[ https://issues.apache.org/jira/browse/HIVE-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622987#comment-15622987 ] Siddharth Seth commented on HIVE-15041: --- [~spena] - should this be committed. Looks like ptest needs a restart fairly often rightnow. (HIVE-15041) > Specify GCE network name on Hive ptest > -- > > Key: HIVE-15041 > URL: https://issues.apache.org/jira/browse/HIVE-15041 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-15041.1.patch > > > NO PRECOMMIT TESTS > A new option on cloudhost.properties should be added to specify the GCE > network name: > # GCE network option > network = > https://www.googleapis.com/compute/v1/projects//global/networks/default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15041) Specify GCE network name on Hive ptest
[ https://issues.apache.org/jira/browse/HIVE-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603171#comment-15603171 ] Sergio Peña commented on HIVE-15041: Sure. I'll do some testing on a ptest environment before commit it. > Specify GCE network name on Hive ptest > -- > > Key: HIVE-15041 > URL: https://issues.apache.org/jira/browse/HIVE-15041 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-15041.1.patch > > > NO PRECOMMIT TESTS > A new option on cloudhost.properties should be added to specify the GCE > network name: > # GCE network option > network = > https://www.googleapis.com/compute/v1/projects//global/networks/default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15041) Specify GCE network name on Hive ptest
[ https://issues.apache.org/jira/browse/HIVE-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603151#comment-15603151 ] Siddharth Seth commented on HIVE-15041: --- +1. [~spena] - could you please run the ptest unit tests before committing (they don't run on jenkins) > Specify GCE network name on Hive ptest > -- > > Key: HIVE-15041 > URL: https://issues.apache.org/jira/browse/HIVE-15041 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-15041.1.patch > > > NO PRECOMMIT TESTS > A new option on cloudhost.properties should be added to specify the GCE > network name: > # GCE network option > network = > https://www.googleapis.com/compute/v1/projects//global/networks/default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
