[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-05-10 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/cloudstack/pull/1466 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-05-09 Thread nvazquez
Github user nvazquez commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-218043423 Thanks @koushik-das, @swill now that there are 2 LGTM can we have this merged? --- If your project is set up for it, you can reply to this email and have your

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-05-05 Thread koushik-das
Github user koushik-das commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-217082822 @serg38 LGTM. Verified on my test env. @swill This is ready for merge. --- If your project is set up for it, you can reply to this email and have your

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-05-04 Thread serg38
Github user serg38 commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-216967912 @rhtyd I just finished QA of changes on MySQL 5.1, 5.5, 5.6 and 5.7. No issues. 64byte index on ssh_keypairs table is created fine on all DB platforms.

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-05-04 Thread rhtyd
Github user rhtyd commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-216910739 Thanks @nvazquez did you test this against mysql 5.1, 5.6 and 5.7? The Travis run will check against MySQL-Server 5.5 --- If your project is set up for it, you can

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-05-04 Thread nvazquez
Github user nvazquez commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-216901961 @koushik-das @swill @serg38 I modified index length for ssh_keypairs table --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-05-04 Thread swill
Github user swill commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-216896697 @serg38, I just saw your communication with @koushik-das. I will let him verify the fix you are working on and we will go from there. Thanks... --- If your

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-05-04 Thread swill
Github user swill commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-216893678 The one failure above is unrelated to this fix and pops up periodically in my environment. Above is a passing CI result... --- If your project is set up for it,

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-05-04 Thread swill
Github user swill commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-216893394 ### CI RESULTS ``` Tests Run: 85 Skipped: 0 Failed: 1 Errors: 0 Duration: 4h 28m 09s ``` **Summary of the

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-05-04 Thread serg38
Github user serg38 commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-216876560 Thanks @koushik-das. Looks like before 5.6 Mysql auto restricted length of InnoDB indexes to 255 characters without giving trouble if index length is not specified.

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-05-03 Thread koushik-das
Github user koushik-das commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-216751340 I am getting this error during deploy DB "Specified key was too long; max key length is 767 bytes". I have provided another inline comment in the code which is

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-05-03 Thread koushik-das
Github user koushik-das commented on a diff in the pull request: https://github.com/apache/cloudstack/pull/1466#discussion_r61992331 --- Diff: setup/db/db/schema-481to490.sql --- @@ -413,3 +413,26 @@ VIEW `user_vm_view` AS -- Add cluster.storage.operations.exclude

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-05-03 Thread swill
Github user swill commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-216576269 I was having problems with this in my CI, but I am still trying to determine if it is my environment or not. I will try again once I get master tested (since we

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-05-03 Thread serg38
Github user serg38 commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-216574386 All checks have been passed after squashing. @swill did you CI run pass? If so can you merge it please? --- If your project is set up for it, you can reply to this

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-05-02 Thread swill
Github user swill commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-216258261 Running CI now... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-05-02 Thread nvazquez
Github user nvazquez commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-216251605 Done, thanks @rhtyd @swill @rafaelweingartner --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-05-02 Thread rhtyd
Github user rhtyd commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-216226707 @nvazquez thanks, can you rebase and squash changes to one commit LGTM, a CI and Travis run should confirm the fix. tag:mergeready --- If your

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-28 Thread swill
Github user swill commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-215586085 OK. I will just do a standard CI run then. Thx. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-28 Thread rafaelweingartner
Github user rafaelweingartner commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-215583632 If you run your CI tests, it should detect if it breaks something. But more importantly would be to check if the performance was improved. But, I am sure

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-28 Thread swill
Github user swill commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-215581686 should I be running some sort of CI or validation for this change? How do I validate this works and more importantly, not break anything? I also need another LGTM.

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-28 Thread rafaelweingartner
Github user rafaelweingartner commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-215572438 off course ;) I was only waiting for that LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-28 Thread serg38
Github user serg38 commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-215572026 With this change PR passed Travis and Jenkins. @rafaelweingartner and @rhtyd will you be able to give your blessing? --- If your project is set up for it, you can

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-28 Thread rafaelweingartner
Github user rafaelweingartner commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-215549970 I confess that without looking the code, I thought the same as @nvazquez. First we would execute schema-XXXtoYYY.sql, schemaXXXtoYYY-cleanup.sql and

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-28 Thread rhtyd
Github user rhtyd commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-215548664 @nvazquez that's by design -- all upgrade paths (migration code and the schema files) are run first and after that only the cleanup scripts are ran --- If your

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-28 Thread nvazquez
Github user nvazquez commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-215548103 @rafaelweingartner exactly, I meant to explain that I realized that first all `schema-XXXtoYYY.sql` were executed and after they finished, all

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-28 Thread rafaelweingartner
Github user rafaelweingartner commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-215546030 @nvazquez, What does that mean? First we are executing all of the `schema-XXXtoYYY.sql`, then all of the `schemaXXXtoYYY-cleanup.sql`?

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-28 Thread nvazquez
Github user nvazquez commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-215529568 @rhtyd @rafaelweingartner I could discover the problem by deleting `cloud` and `cloud_usage` database, then `cloudstack-databases-setup` to create them blank and

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-26 Thread rafaelweingartner
Github user rafaelweingartner commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-214861439 On Jenkins I saw an error that is due to environment misconfigurations: > [FATAL] Non-parseable settings /home/jenkins/.m2/settings.xml:

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-26 Thread rhtyd
Github user rhtyd commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-214856702 @nvazquez I'm not sure, can you try to reproduce and fix it locally; test with MySQL 5.6 and 5.5 (both major versions) --- If your project is set up for it, you can

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-26 Thread nvazquez
Github user nvazquez commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-214831953 @rafaelweingartner @rhtyd @swill I did `push -f` tests start again, I'm getting an error on Travis test due to duplicate field name 'size' in

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-11 Thread nvazquez
Github user nvazquez commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-208402795 Thanks @serg38 @rafaelweingartner. I rebased master branch and pushed again --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-11 Thread rafaelweingartner
Github user rafaelweingartner commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-208393777 @serg38 thanks, now it is clear to me. Thanks for the great job. BTW: it seems that travis is with problems again. --- If your project is set up

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-11 Thread serg38
Github user serg38 commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-208387324 @rafaelweingartner. That's correct. 3 indexes: 1 - primary + 2 unique indexes, all based on ID column. Those 2 unique indexes are not needed and only consume disk

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-11 Thread rafaelweingartner
Github user rafaelweingartner commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-208285159 Thanks @serg38, You answer helped a lot. I just do not fully understand the deletion of the indexes, for instance, you delete index “id” and

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-10 Thread serg38
Github user serg38 commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-208121230 @rafaelweingartner Here you change this table "ovs_tunnel_network", you dropped the primary key "id"; Will that remove only the index or the field

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-10 Thread rafaelweingartner
Github user rafaelweingartner commented on a diff in the pull request: https://github.com/apache/cloudstack/pull/1466#discussion_r59147080 --- Diff: setup/db/db/schema-481to490.sql --- @@ -413,3 +413,89 @@ VIEW `user_vm_view` AS -- Add cluster.storage.operations.exclude

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-10 Thread rafaelweingartner
Github user rafaelweingartner commented on a diff in the pull request: https://github.com/apache/cloudstack/pull/1466#discussion_r59146919 --- Diff: setup/db/db/schema-481to490.sql --- @@ -413,3 +413,89 @@ VIEW `user_vm_view` AS -- Add cluster.storage.operations.exclude

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-10 Thread rafaelweingartner
Github user rafaelweingartner commented on a diff in the pull request: https://github.com/apache/cloudstack/pull/1466#discussion_r59146860 --- Diff: setup/db/db/schema-481to490.sql --- @@ -413,3 +413,89 @@ VIEW `user_vm_view` AS -- Add cluster.storage.operations.exclude

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-10 Thread rafaelweingartner
Github user rafaelweingartner commented on a diff in the pull request: https://github.com/apache/cloudstack/pull/1466#discussion_r59146682 --- Diff: setup/db/db/schema-481to490.sql --- @@ -413,3 +413,89 @@ VIEW `user_vm_view` AS -- Add cluster.storage.operations.exclude

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-10 Thread nvazquez
Github user nvazquez commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-207994438 @bhaisaab Most of the indexes being dropped are duplicate. In create-schema.sql quite a lot of tables are created like this: CREATE TABLE `cloud`.`version` (

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-09 Thread bhaisaab
Github user bhaisaab commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-207824671 @nvazquez can you explain how did you find out which keys (i.e indexes) to drop? --- If your project is set up for it, you can reply to this email and have your

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-09 Thread nvazquez
Github user nvazquez commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-207823632 Thanks @rafaelweingartner, @swill and @bhaisaab for your help. I pushed changes again --- If your project is set up for it, you can reply to this email and have

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-08 Thread swill
Github user swill commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-207624970 Thank you very much for the quick follow up on this, it is greatly appreciated. :+1: I will review the linked PR. --- If your project is set up for it, you can

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-08 Thread bhaisaab
Github user bhaisaab commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-207617920 @swill @nvazquez I saw the failing job: https://travis-ci.org/apache/cloudstack/jobs/121766436 It failed due to missing Marvin distribution: IOError:

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-08 Thread swill
Github user swill commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-207591089 I saw travis failing with this earlier. I will see if I can figure out what is going on. Thx... --- If your project is set up for it, you can reply to this email

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-08 Thread nvazquez
Github user nvazquez commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-207562190 You're welcome :) Now Travis failed without including changes (only comment line). Do I try again after Jenkins finishes using push -f? --- If your project

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-08 Thread swill
Github user swill commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-207551688 Oh, I understand. Thanks... :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-08 Thread nvazquez
Github user nvazquez commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-207550783 Thanks @swill, I just wanted to remove all my commits for Travis test to run without my changes, but doing that closed the pr. Now I pushed a commit including a

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-08 Thread swill
Github user swill commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-207537254 Just for the future reference. If you want to kick off the rebuild of a PR, you can do a `push -f` which will trigger a new build even if the code did not change in

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-08 Thread nvazquez
Github user nvazquez commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-207535135 @rafaelweingartner yes, it was my mistake. I wanted to leave the pr with no changes to test if Travis passes, I didn't know that removing my commits will close

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-08 Thread nvazquez
GitHub user nvazquez reopened a pull request: https://github.com/apache/cloudstack/pull/1466 CLOUDSTACK-9340: General DB Optimization ## Description In some production environments there were being experimented delays in most of the jobs. A search for DB optimization was taken

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-08 Thread rafaelweingartner
Github user rafaelweingartner commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-207534182 are you still going to push this forward? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-08 Thread nvazquez
Github user nvazquez closed the pull request at: https://github.com/apache/cloudstack/pull/1466 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-08 Thread nvazquez
Github user nvazquez commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-207498927 Yes, this test failed, but I didn't see schema error this time === TestName: test_listVM_by_id_as_user_vmsfromotherdomain | Status : EXCEPTION ===

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-08 Thread rafaelweingartner
Github user rafaelweingartner commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-207473461 Now the error is different, right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-08 Thread nvazquez
Github user nvazquez commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-207472873 @rafaelweingartner sorry I pushed force again, after the whole file failed I pushed a part of sql file and pushed it and started again --- If your project is set

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-08 Thread rafaelweingartner
Github user rafaelweingartner commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-207471025 @nvazquez did it fail with the problem? for me it is showing as executing and not failed. --- If your project is set up for it, you can reply to this

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-08 Thread rafaelweingartner
Github user rafaelweingartner commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-207440848 @nvazquez, it is complain about a column called "size" at the "schema-40to410.sql" > schema-40to410.sql: Duplicate column name 'size'

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-08 Thread nvazquez
Github user nvazquez commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-207434839 Hi @rafaelweingartner I executed sql in my env and succeeded, is there a way to trace which line failed? --- If your project is set up for it, you can reply to

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-07 Thread nvazquez
Github user nvazquez commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-207081102 @rafaelweingartner I'll check this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-07 Thread rafaelweingartner
Github user rafaelweingartner commented on the pull request: https://github.com/apache/cloudstack/pull/1466#issuecomment-207057213 @nvazquez there is something wrong with one of your SQLs. > Unable to upgrade the database: Unable to execute upgrade script:

[GitHub] cloudstack pull request: CLOUDSTACK-9340: General DB Optimization

2016-04-07 Thread nvazquez
GitHub user nvazquez opened a pull request: https://github.com/apache/cloudstack/pull/1466 CLOUDSTACK-9340: General DB Optimization ## Description In some production environments there were being experimented delays in most of the jobs. A search for DB optimization was taken