[ovirt-devel] Re: maven central brings some surprises

[Roman Mohr]

On Fri, Jan 17, 2020 at 11:24 AM Fedor Gavrilov  wrote:

> Hi,
>
> Maven central seems to be not in the mood today for me:
>
> [ERROR] Plugin org.apache.maven.plugins:maven-jar-plugin:2.2 or one of its
> dependencies could not be resolved: Failed to read artifact descriptor for
> org.apache.maven.plugins:maven-jar-plugin:jar:2.2: Could not transfer
> artifact org.apache.maven.plugins:maven-jar-plugin:pom:2.2 from/to central (
> http://repo1.maven.org/maven2): Failed to transfer file:
> http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-jar-plugin/2.2/maven-jar-plugin-2.2.pom.
> Return code is: 501 , ReasonPhrase:HTTPS Required. -> [Help 1]
>
>
It now only accepts tls connections. Try with https.

Best Regards,
Roman


> Does anyone know if that's somehow a proxy thing or something else? I'm
> asking because I had a joy of dealing with lying proxy before.
> Also, iirc jcenter had a reputation of being more reliable, can we use it
> instead somehow?
>
> Thanks,
> Fedor
> ___
> Devel mailing list -- devel@ovirt.org
> To unsubscribe send an email to devel-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/SM463BM35XSOVAS3LNEAA37W6YFNKASD/
>
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3YI4VLTL52XWRNSLGUKDQJXDGDCWCJSL/

[ovirt-devel] Re: [Fedora][Engine] Hystrix going to be orphaned in Fedora

[Roman Mohr]

On Tue, May 29, 2018 at 11:11 AM, Roman Mohr  wrote:

>
>
> On Tue, May 29, 2018 at 11:00 AM, Sandro Bonazzola 
> wrote:
>
>>
>>
>> 2018-05-29 10:57 GMT+02:00 Roman Mohr :
>>
>>>
>>>
>>> On Tue, May 29, 2018 at 10:43 AM, Roman Mohr  wrote:
>>>
>>>>
>>>>
>>>> On Tue, May 29, 2018 at 10:06 AM, Sandro Bonazzola >>> > wrote:
>>>>
>>>>> FESCo approved an updated policy for packages which fail to build from
>>>>> source during mass rebuilds (FTBFS).
>>>>>
>>>>> The updated policy is still at https://fedoraproject.org/w
>>>>> iki/Fails_to_build_from_source.
>>>>>
>>>>> Highlights:
>>>>>
>>>>> - packages which FTBFS are subject to orphaning if there is no
>>>>>   maintainer acknowledgement within 8 weeks
>>>>>
>>>>> - packages which FTBFS in two consecutive mass rebuilds will be
>>>>>   retired soon after the second mass rebuild
>>>>>
>>>>> Focusing on Hystrix, it has been reported failing on 2018-03-14, more
>>>>> than 10 weeks ago  and it's failing to build since Fedora 26, totalling 3
>>>>> consecutive mass rebuild failures.
>>>>> This put Hystrix in a candidate position for being orphaned.
>>>>>
>>>>> The Hystrix package is used by oVirt Engine.
>>>>> The POM requires version 1.4.14 (released this on 6 Aug 2015) .
>>>>> Fedora 24 and CentOS Virt SIG has 1.4.21 (released this on 17 Nov
>>>>> 2015).
>>>>> Latest upstream version is 1.5.13.
>>>>>
>>>>> Since in oVirt 4.3 we're aiming to support Fedora 28 we need to decide
>>>>> on what to do with Hystrix:
>>>>> - drop it as dependency?
>>>>> - update to latest?
>>>>> - try to fix build failure and keep current version?
>>>>>
>>>>> Roman, current maintainer (in cc) is unresponsive, not replying to my
>>>>> needinfo since 2017-08-17
>>>>>
>>>>
>>>>
>>>> I simply missed it. At least a concrete ping would have been nice.
>>>>
>>>>
>>> Ok, my failure. I got a ping.
>>>
>>
>> NP, I'm happy you're not MIA :-)
>> What's your view on this topic?
>>
>>
> I am updating it right now to 1.5.13. I have no strong opinions. If oVirt
> will not use it anymore there is probably not much reason to keep it then.
> I don't think that it has wide adoption.
>


An update here: Hystrix 1.5.12 is in rawhide working again since last week.
The update for fc28 can be found in [1].

Best Regards,

Roman

[1] https://bodhi.fedoraproject.org/updates/FEDORA-2018-a1647d6783


>
> Best Regards,
>
> Roman
>
>
>>
>>
>>>
>>>
>>>
>>>> Best Regards,
>>>>
>>>> Roman
>>>>
>>>>
>>>>>
>>>>> Any volunteer for taking over?
>>>>>
>>>>> --
>>>>>
>>>>> SANDRO BONAZZOLA
>>>>>
>>>>> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R
>>>>>
>>>>> Red Hat EMEA <https://www.redhat.com/>
>>>>>
>>>>> sbona...@redhat.com
>>>>> <https://red.ht/sig>
>>>>> <https://redhat.com/summit>
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>>
>> SANDRO BONAZZOLA
>>
>> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R
>>
>> Red Hat EMEA <https://www.redhat.com/>
>>
>> sbona...@redhat.com
>> <https://red.ht/sig>
>> <https://redhat.com/summit>
>>
>
>
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/LSE4KIAE24VPOPSWOALHLPGPNLQLLFIT/

[ovirt-devel] Re: [Fedora][Engine] Hystrix going to be orphaned in Fedora

[Roman Mohr]

On Tue, May 29, 2018 at 11:00 AM, Sandro Bonazzola 
wrote:

>
>
> 2018-05-29 10:57 GMT+02:00 Roman Mohr :
>
>>
>>
>> On Tue, May 29, 2018 at 10:43 AM, Roman Mohr  wrote:
>>
>>>
>>>
>>> On Tue, May 29, 2018 at 10:06 AM, Sandro Bonazzola 
>>> wrote:
>>>
>>>> FESCo approved an updated policy for packages which fail to build from
>>>> source during mass rebuilds (FTBFS).
>>>>
>>>> The updated policy is still at https://fedoraproject.org/w
>>>> iki/Fails_to_build_from_source.
>>>>
>>>> Highlights:
>>>>
>>>> - packages which FTBFS are subject to orphaning if there is no
>>>>   maintainer acknowledgement within 8 weeks
>>>>
>>>> - packages which FTBFS in two consecutive mass rebuilds will be
>>>>   retired soon after the second mass rebuild
>>>>
>>>> Focusing on Hystrix, it has been reported failing on 2018-03-14, more
>>>> than 10 weeks ago  and it's failing to build since Fedora 26, totalling 3
>>>> consecutive mass rebuild failures.
>>>> This put Hystrix in a candidate position for being orphaned.
>>>>
>>>> The Hystrix package is used by oVirt Engine.
>>>> The POM requires version 1.4.14 (released this on 6 Aug 2015) .
>>>> Fedora 24 and CentOS Virt SIG has 1.4.21 (released this on 17 Nov 2015).
>>>> Latest upstream version is 1.5.13.
>>>>
>>>> Since in oVirt 4.3 we're aiming to support Fedora 28 we need to decide
>>>> on what to do with Hystrix:
>>>> - drop it as dependency?
>>>> - update to latest?
>>>> - try to fix build failure and keep current version?
>>>>
>>>> Roman, current maintainer (in cc) is unresponsive, not replying to my
>>>> needinfo since 2017-08-17
>>>>
>>>
>>>
>>> I simply missed it. At least a concrete ping would have been nice.
>>>
>>>
>> Ok, my failure. I got a ping.
>>
>
> NP, I'm happy you're not MIA :-)
> What's your view on this topic?
>
>
I am updating it right now to 1.5.13. I have no strong opinions. If oVirt
will not use it anymore there is probably not much reason to keep it then.
I don't think that it has wide adoption.

Best Regards,

Roman


>
>
>>
>>
>>
>>> Best Regards,
>>>
>>> Roman
>>>
>>>
>>>>
>>>> Any volunteer for taking over?
>>>>
>>>> --
>>>>
>>>> SANDRO BONAZZOLA
>>>>
>>>> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R
>>>>
>>>> Red Hat EMEA <https://www.redhat.com/>
>>>>
>>>> sbona...@redhat.com
>>>> <https://red.ht/sig>
>>>> <https://redhat.com/summit>
>>>>
>>>
>>>
>>
>
>
> --
>
> SANDRO BONAZZOLA
>
> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R
>
> Red Hat EMEA <https://www.redhat.com/>
>
> sbona...@redhat.com
> <https://red.ht/sig>
> <https://redhat.com/summit>
>
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/TTHPETX6YFT6WGTDA6SQDPI3T4UKLEB7/

[ovirt-devel] Re: [Fedora][Engine] Hystrix going to be orphaned in Fedora

[Roman Mohr]

On Tue, May 29, 2018 at 10:43 AM, Roman Mohr  wrote:

>
>
> On Tue, May 29, 2018 at 10:06 AM, Sandro Bonazzola 
> wrote:
>
>> FESCo approved an updated policy for packages which fail to build from
>> source during mass rebuilds (FTBFS).
>>
>> The updated policy is still at https://fedoraproject.org/w
>> iki/Fails_to_build_from_source.
>>
>> Highlights:
>>
>> - packages which FTBFS are subject to orphaning if there is no
>>   maintainer acknowledgement within 8 weeks
>>
>> - packages which FTBFS in two consecutive mass rebuilds will be
>>   retired soon after the second mass rebuild
>>
>> Focusing on Hystrix, it has been reported failing on 2018-03-14, more
>> than 10 weeks ago  and it's failing to build since Fedora 26, totalling 3
>> consecutive mass rebuild failures.
>> This put Hystrix in a candidate position for being orphaned.
>>
>> The Hystrix package is used by oVirt Engine.
>> The POM requires version 1.4.14 (released this on 6 Aug 2015) .
>> Fedora 24 and CentOS Virt SIG has 1.4.21 (released this on 17 Nov 2015).
>> Latest upstream version is 1.5.13.
>>
>> Since in oVirt 4.3 we're aiming to support Fedora 28 we need to decide on
>> what to do with Hystrix:
>> - drop it as dependency?
>> - update to latest?
>> - try to fix build failure and keep current version?
>>
>> Roman, current maintainer (in cc) is unresponsive, not replying to my
>> needinfo since 2017-08-17
>>
>
>
> I simply missed it. At least a concrete ping would have been nice.
>
>
Ok, my failure. I got a ping.



> Best Regards,
>
> Roman
>
>
>>
>> Any volunteer for taking over?
>>
>> --
>>
>> SANDRO BONAZZOLA
>>
>> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R
>>
>> Red Hat EMEA <https://www.redhat.com/>
>>
>> sbona...@redhat.com
>> <https://red.ht/sig>
>> <https://redhat.com/summit>
>>
>
>
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/SFHVZ4HO5ERYGSYXRYCL432GPUN7C4N4/

[ovirt-devel] Re: [Fedora][Engine] Hystrix going to be orphaned in Fedora

[Roman Mohr]

On Tue, May 29, 2018 at 10:06 AM, Sandro Bonazzola 
wrote:

> FESCo approved an updated policy for packages which fail to build from
> source during mass rebuilds (FTBFS).
>
> The updated policy is still at https://fedoraproject.org/w
> iki/Fails_to_build_from_source.
>
> Highlights:
>
> - packages which FTBFS are subject to orphaning if there is no
>   maintainer acknowledgement within 8 weeks
>
> - packages which FTBFS in two consecutive mass rebuilds will be
>   retired soon after the second mass rebuild
>
> Focusing on Hystrix, it has been reported failing on 2018-03-14, more than
> 10 weeks ago  and it's failing to build since Fedora 26, totalling 3
> consecutive mass rebuild failures.
> This put Hystrix in a candidate position for being orphaned.
>
> The Hystrix package is used by oVirt Engine.
> The POM requires version 1.4.14 (released this on 6 Aug 2015) .
> Fedora 24 and CentOS Virt SIG has 1.4.21 (released this on 17 Nov 2015).
> Latest upstream version is 1.5.13.
>
> Since in oVirt 4.3 we're aiming to support Fedora 28 we need to decide on
> what to do with Hystrix:
> - drop it as dependency?
> - update to latest?
> - try to fix build failure and keep current version?
>
> Roman, current maintainer (in cc) is unresponsive, not replying to my
> needinfo since 2017-08-17
>


I simply missed it. At least a concrete ping would have been nice.

Best Regards,

Roman


>
> Any volunteer for taking over?
>
> --
>
> SANDRO BONAZZOLA
>
> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R
>
> Red Hat EMEA 
>
> sbona...@redhat.com
> 
> 
>
___
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/LAAIVZEJJ6D2ERNOZXT52N44HSHHKPRH/

Re: [ovirt-devel] Fwd: Still Failing: oVirt/ovirt-engine#4888 (master - 1cc377f)

[Roman Mohr]

On Tue, Mar 6, 2018 at 9:34 AM, Roman Mohr <rm...@redhat.com> wrote:

>
>
> On Tue, Mar 6, 2018 at 8:11 AM, Sandro Bonazzola <sbona...@redhat.com>
> wrote:
>
>> Hi,
>> not sure who's monitoring Travis failures and maintaining Travis jobs,
>>
>
> I fear that was me until I moved to KubeVirt and that currently no one
> monitors it. It is mostly useful to generate the sonarqube code-analysis.
>
>
>> but looks like we have an error in the test suite
>>
>
>
> I guess this is related to the postgresql database version provided to
> travis. See https://travis-ci.org/oVirt/ovirt-engine/builds/349675802#L497.
> It complains about:
>
> psql:./packaging/dbscripts/common_sp.sql:1305: ERROR: type jsonb does not
> exist
>
> Is postgres 9.2 still correct, or is there something needed to enable
> jsonb? See https://github.com/oVirt/ovirt-engine/blob/master/.
> travis.yml#L17 for the current version used.
>
> To be honest I am not sure if that free sonarqube offer still exists. You
> can probably just remove the .travis.yml file.
>


Oh nemo.sonarqube.org is still free for opensource project, they just
changed the mein url:
https://sonarcloud.io/dashboard?id=org.ovirt.engine%3Aroot

So it might still be useful. There is this other code analysis too, so up
to you guys if you find it useful.


>
> Best Regards,
>
> Roman
>
>
>> It's currently failing on:
>>
>>   ---
>>  T E S T S
>> ---
>> Running org.ovirt.engine.core.dal.job.ExecutionMessageDirectorTest
>> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.234 sec
>> Running org.ovirt.engine.core.dal.dbbroker.DbConnectionUtilTest
>> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.936 sec 
>> <<< FAILURE!
>> org.ovirt.engine.core.dal.dbbroker.DbConnectionUtilTest  Time elapsed: 936 
>> sec  <<< FAILURE!
>> java.lang.AssertionError: Unable to init tests
>>
>> Looks pretty much similar to what happens in jenkins for fcraw builds,
>> tracked here: https://bugzilla.redhat.com/show_bug.cgi?id=1550033
>>
>>
>>
>> -- Forwarded message --
>> From: Travis CI <bui...@travis-ci.org>
>> Date: 2018-03-06 7:53 GMT+01:00
>> Subject: Still Failing: oVirt/ovirt-engine#4888 (master - 1cc377f)
>> To: sbona...@redhat.com
>>
>>
>> *oVirt / ovirt-engine
>> <https://travis-ci.org/oVirt/ovirt-engine?utm_source=email_medium=notification>*
>> (master <https://github.com/oVirt/ovirt-engine/tree/master>)
>> Build #4888 is still failing.
>> <https://travis-ci.org/oVirt/ovirt-engine/builds/349675802?utm_source=email_medium=notification>
>> 4 minutes and 38 seconds
>> *Yedidyah Bar David* 1cc377f
>> <https://github.com/oVirt/ovirt-engine/commit/1cc377fa30286733aeb68aa75a5bc801d6fc3f41>
>>  Changeset
>> →
>> <https://github.com/oVirt/ovirt-engine/compare/6c9465241694...1cc377fa3028>
>>   packaging: setup: postgres95: Refuse to upgrade if new data dir is not
>> empty
>>
>> Change-Id: I9f69ceb2b8ea773d1bb883c3e81734a3c4c8af3c
>> Bug-Url: https://bugzilla.redhat.com/1546771
>> Signed-off-by: Yedidyah Bar David <d...@redhat.com>
>>
>> *Want to know about upcoming build environment updates?*
>>
>> Would you like to stay up-to-date with the upcoming Travis CI build
>> environment updates? We set up a mailing list for you! Sign up here
>> <http://eepurl.com/9OCsP>.
>> Documentation <https://docs.travis-ci.com> about Travis CI
>> Need help? Mail support <supp...@travis-ci.com>!
>> Choose who receives these build notification emails in your configuration
>> file <https://docs.travis-ci.com/user/notifications>.
>>
>> *Would you like to test your private code?*
>>
>> Travis CI for Private Projects
>> <https://travis-ci.com?utm_source=build_email_footer_campaign=travis-ci.org_medium=email>
>> could be your new best friend!
>>
>>
>>
>> --
>>
>> SANDRO BONAZZOLA
>>
>> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R
>>
>> Red Hat EMEA <https://www.redhat.com/>
>> <https://red.ht/sig>
>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>>
>>
>> ___
>> Devel mailing list
>> Devel@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Fwd: Still Failing: oVirt/ovirt-engine#4888 (master - 1cc377f)

[Roman Mohr]

On Tue, Mar 6, 2018 at 8:11 AM, Sandro Bonazzola 
wrote:

> Hi,
> not sure who's monitoring Travis failures and maintaining Travis jobs,
>

I fear that was me until I moved to KubeVirt and that currently no one
monitors it. It is mostly useful to generate the sonarqube code-analysis.


> but looks like we have an error in the test suite
>


I guess this is related to the postgresql database version provided to
travis. See https://travis-ci.org/oVirt/ovirt-engine/builds/349675802#L497.
It complains about:

psql:./packaging/dbscripts/common_sp.sql:1305: ERROR: type jsonb does not
exist

Is postgres 9.2 still correct, or is there something needed to enable
jsonb? See https://github.com/oVirt/ovirt-engine/blob/master/.travis.yml#L17
for the current version used.

To be honest I am not sure if that free sonarqube offer still exists. You
can probably just remove the .travis.yml file.

Best Regards,

Roman


> It's currently failing on:
>
>   ---
>  T E S T S
> ---
> Running org.ovirt.engine.core.dal.job.ExecutionMessageDirectorTest
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.234 sec
> Running org.ovirt.engine.core.dal.dbbroker.DbConnectionUtilTest
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.936 sec <<< 
> FAILURE!
> org.ovirt.engine.core.dal.dbbroker.DbConnectionUtilTest  Time elapsed: 936 
> sec  <<< FAILURE!
> java.lang.AssertionError: Unable to init tests
>
> Looks pretty much similar to what happens in jenkins for fcraw builds,
> tracked here: https://bugzilla.redhat.com/show_bug.cgi?id=1550033
>
>
>
> -- Forwarded message --
> From: Travis CI 
> Date: 2018-03-06 7:53 GMT+01:00
> Subject: Still Failing: oVirt/ovirt-engine#4888 (master - 1cc377f)
> To: sbona...@redhat.com
>
>
> *oVirt / ovirt-engine
> *
> (master )
> Build #4888 is still failing.
> 
> 4 minutes and 38 seconds
> *Yedidyah Bar David* 1cc377f
> 
>  Changeset
> →
> 
>   packaging: setup: postgres95: Refuse to upgrade if new data dir is not
> empty
>
> Change-Id: I9f69ceb2b8ea773d1bb883c3e81734a3c4c8af3c
> Bug-Url: https://bugzilla.redhat.com/1546771
> Signed-off-by: Yedidyah Bar David 
>
> *Want to know about upcoming build environment updates?*
>
> Would you like to stay up-to-date with the upcoming Travis CI build
> environment updates? We set up a mailing list for you! Sign up here
> .
> Documentation  about Travis CI
> Need help? Mail support !
> Choose who receives these build notification emails in your configuration
> file .
>
> *Would you like to test your private code?*
>
> Travis CI for Private Projects
> 
> could be your new best friend!
>
>
>
> --
>
> SANDRO BONAZZOLA
>
> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R
>
> Red Hat EMEA 
> 
> TRIED. TESTED. TRUSTED. 
>
>
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Toward self-configuring CI, or how can we stop writing YAML

[Roman Mohr]

On Tue, Jan 10, 2017 at 2:28 PM, Barak Korren  wrote:

> >>
> >> Oh, but it does.. you can configure publishers ans webhooks in the
> >> .travis.yml file. I commonly build a project, create a docker image
> >> using the built bits and deploy to my VPS (using a branch name check).
> >>
> >
> > What travis does not have is build chains in the sense of multiple jobs
> by
> > different yaml files, but as Martin says the rest is there. What they
> have
> > is different steps inside one yaml file (setup, pre_script, post_script,
> > deployment, ...)
> > which can be triggered according to branches and tags.
> >
> > In Kubevirt we also use a .travis.yaml which does testing and releasing
> > based on tags and branches [1].
> >
> > Note that for more finetuned controls, they also provide a lot of
> > environment variables which you can reference (e.g. branch, tag, ...)
> >
>
> Lets not run this into an off-topic 'travis vs. foo' discussion. This
> is not going anywhere.
>
>
Not trying to make any point, except from providing insight how travis is
doing it. Since it might be useful.



> --
> Barak Korren
> bkor...@redhat.com
> RHCE, RHCi, RHV-DevOps Team
> https://ifireball.wordpress.com/
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Toward self-configuring CI, or how can we stop writing YAML

[Roman Mohr]

On Tue, Jan 10, 2017 at 2:18 PM, Martin Sivak  wrote:

> > While packages are separate entities and have independent life cycles,
> > we still need a way to compose and test oVirt as a whole. And the
> > responsibility to determine which version of the package goes into
> > which oVirt release should, in many cases lay on the shoulders of that
> > package maintainer.
> >
> > Having distgit-like solution can be simple enough, but it sounds to me
> > like an overkill in many cases.
> >
> > How would you suggest to solve this?
>
> Well, what about another text file in the automation directory?
> use-for.txt with ovirt-x.y and wildcards?
>
> > Travis never concerns itself with full system composition and
> > integration testing, so this is not a useful analogy at all.
>
> Oh, but it does.. you can configure publishers ans webhooks in the
> .travis.yml file. I commonly build a project, create a docker image
> using the built bits and deploy to my VPS (using a branch name check).
>
>
What travis does not have is build chains in the sense of multiple jobs by
different yaml files, but as Martin says the rest is there. What they have
is different steps inside one yaml file (setup, pre_script, post_script,
deployment, ...)
which can be triggered according to branches and tags.

In Kubevirt we also use a .travis.yaml which does testing and releasing
based on tags and branches [1].

Note that for more finetuned controls, they also provide a lot of
environment variables which you can reference (e.g. branch, tag, ...)

Best Regards,

Roman


> Martin
>
> On Tue, Jan 10, 2017 at 1:06 PM, Barak Korren  wrote:
> > On 10 January 2017 at 12:22, Martin Sivak  wrote:
> >> Hi,
> >>
> >> all of that makes sense except this:
> >>
> >>> When it comes to branches, I think the way to go is to standardize
> >>> branch names. That standard should probably be something like
> >>> 'ovirt-4.0', 'ovirt-4.1', etc. Alternatively we could use something
> >>> like 'ovirt(-.*)?-.4.0' or (-4.0) to accommodate existing
> >>> conventions like the engine`s.
> >>
> >> Do not force a branching model on me, please. We have small packages
> >> that either do not need any branches at all or we create the Z stream
> >> branch on as needed basis only. For example: mom does not need
> >> branches it is always compatible, ovirt-optimizer ever had only one Z
> >> stream patch in its history...
> >>
> >> Please start treating packages as separate entities with independent
> >> development space. You are trying to workaround the lack of distgit by
> >> forcing us to play with our upstream source code repository. You
> >> should stop doing that as you will get into trouble once a first
> >> non-gerrit project is accepted to oVirt. (And we have couple of
> >> project where GitHub is the primary platform already)
> >
> > While packages are separate entities and have independent life cycles,
> > we still need a way to compose and test oVirt as a whole. And the
> > responsibility to determine which version of the package goes into
> > which oVirt release should, in many cases lay on the shoulders of that
> > package maintainer.
> >
> > Having distgit-like solution can be simple enough, but it sounds to me
> > like an overkill in many cases.
> >
> > How would you suggest to solve this?
> >
> >> The way this is done on Travis for example is much better. Pushes to
> >> all branches are monitored and branches with no .travis.yml (or
> >> automation dir in our case) are ignored.
> >
> > Travis never concerns itself with full system composition and
> > integration testing, so this is not a useful analogy at all.
> >
> >
> > --
> > Barak Korren
> > bkor...@redhat.com
> > RHCE, RHCi, RHV-DevOps Team
> > https://ifireball.wordpress.com/
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>

[1] https://github.com/kubevirt/kubevirt/blob/master/.travis.yml
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] The feature everyone was asking for is finally here...

[Roman Mohr]

On Wed, Jan 4, 2017 at 4:31 PM, Eyal Edri  wrote:

> FYI,
>
> After many requests from multiple developers and testers, the oVirt CI
> added a new simple job that lets you run the full fledged end-to-end oVirt
> system tests with a click of a button.
> You can read all the details and how-to in the new oVirt blog [1].
>
> We wanted to allow running oVirt system tests on EVERY open patch from ANY
> oVirt project, without relaying on complex building code inside the job.
> Luckily we just added the 'build-on-demand' so together with it you can
> build any rpms you'd like and use them to run the manual job.
>
> So the 2 steps you'll need to do are:
>
>1. Write 'ci please build' inside a comment on an open oVirt patch (
>make sure the feature is enabled for that project first, its already
>available for ovirt-engine,vdsm,dashboard and vdsm-jsonrpc-java)
>2. Run the manual OST job for the version you'd like to test with the
>URLs you got from #1
>
> You'll get and email once the job is done and you can browse the results
> and check for logs from engine and the hosts.
>
> Please feel free to ask questions on in...@ovirt.org as usual.
>
>
> [1] https://www.ovirt.org/blog/2017/01/ovirt-system-tests-to-the-rescue/
>

Very nice!


> --
> Eyal Edri
> Associate Manager
> RHV DevOps
> EMEA ENG Virtualization R
> Red Hat Israel
>
> phone: +972-9-7692018 <+972%209-769-2018>
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Lowering the bar for wiki contribution?

[Roman Mohr]

On Wed, Jan 4, 2017 at 8:57 AM, Roy Golan  wrote:

> I'm getting the feeling I'm not alone in this, authoring and publishing a
> wiki page isn't as used to be for long time.
>
> I want to suggest a bit lighter workflow:
>
> 1.  Everyone can merge their page - (it's a wiki)
>   Same as with (public and open) code, no one has the motivation to
> publish a badly written
>   wiki page under their name. True, it can have an impact, but not as with
> broken code
>
> 2. Use Page-Status marker
>  The author first merges the draft. Its now out there and should be
> updated as time goes and its
>  status is DRAFT. Maintainers will come later and after review would
> change the status to
>  PUBLISH. That could be a header in on the page:
>  ---
>  page status: DRAFT/PUBLISH
>  ---
>
>  Simple I think, and should work.
>
>
Sounds very good.

+1


>
>
>
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

[ovirt-devel] Upgrade from 3.6 el7 job broken?

[Roman Mohr]

Hi,

I saw a lot of failing builds lately on this job:


http://jenkins.ovirt.org/job/ovirt-engine_master_upgrade-from-3.6_el7_created/

One log which still exists is:

http://jenkins.ovirt.org/job/ovirt-engine_master_upgrade-from-3.6_el7_created/3666/
It seems like the BUILD ENGINE RPM step is failing, but I can't see any
reason why:

BUILDING ENGINE RPM
+ create_rpms 
/home/jenkins/workspace/ovirt-engine_master_upgrade-from-3.6_el7_created/tmp_repo/ovirt-engine-4.1.0-0.0.master.20161014001903.el7.centos.src.rpm
/home/jenkins/workspace/ovirt-engine_master_upgrade-from-3.6_el7_created/tmp_repo
.gitee47dd2
+ local 
src_rpm=/home/jenkins/workspace/ovirt-engine_master_upgrade-from-3.6_el7_created/tmp_repo/ovirt-engine-4.1.0-0.0.master.20161014001903.el7.centos.src.rpm
+ local 
dst_dir=/home/jenkins/workspace/ovirt-engine_master_upgrade-from-3.6_el7_created/tmp_repo
+ local release=.gitee47dd2
+ local 
workspace=/home/jenkins/workspace/ovirt-engine_master_upgrade-from-3.6_el7_created
+ local 'BUILD_JAVA_OPTS_MAVEN=-XX:MaxPermSize=1G
-Dgwt.compiler.localWorkers=1 '
+ local 'BUILD_JAVA_OPTS_GWT=-XX:PermSize=512M
-XX:MaxPermSize=1G -Xms1G -Xmx6G '
+ env 'BUILD_JAVA_OPTS_MAVEN=-XX:MaxPermSize=1G
-Dgwt.compiler.localWorkers=1 ' 'BUILD_JAVA_OPTS_GWT=
-XX:PermSize=512M -XX:MaxPermSize=1G -Xms1G -Xmx6G '
rpmbuild -D 'ovirt_build_minimal 1' -D 'release_suffix .gitee47dd2' -D
'ovirt_build_extra_flags -gs
/home/jenkins/workspace/ovirt-engine_master_upgrade-from-3.6_el7_created/artifactory-ovirt-org-settings.xml'
-D '_srcrpmdir 
/home/jenkins/workspace/ovirt-engine_master_upgrade-from-3.6_el7_created/tmp_repo'
-D '_specdir 
/home/jenkins/workspace/ovirt-engine_master_upgrade-from-3.6_el7_created/tmp_repo'
-D '_sourcedir 
/home/jenkins/workspace/ovirt-engine_master_upgrade-from-3.6_el7_created/tmp_repo'
-D '_rpmdir 
/home/jenkins/workspace/ovirt-engine_master_upgrade-from-3.6_el7_created/tmp_repo'
-D '_builddir 
/home/jenkins/workspace/ovirt-engine_master_upgrade-from-3.6_el7_created/tmp_repo'
--rebuild 
/home/jenkins/workspace/ovirt-engine_master_upgrade-from-3.6_el7_created/tmp_repo/ovirt-engine-4.1.0-0.0.master.20161014001903.el7.centos.src.rpm
+ return 1
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for :.* : True
Logical operation result is TRUE
Running script  : #!/bin/bash -x


Best Regards,


Roman
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Proposing Martin Sivak as a Backend Maintainer for SLA & Scheduling

[Roman Mohr]

On Mon, Sep 19, 2016 at 4:05 PM, Martin Betak  wrote:

>
>
>
>
> - Original Message -
> > From: "Arik Hadas" 
> > To: "Doron Fediuck" 
> > Cc: "devel" 
> > Sent: Monday, September 19, 2016 3:52:53 PM
> > Subject: Re: [ovirt-devel] Proposing Martin Sivak as a Backend
> Maintainer for SLA & Scheduling
> >
> >
> >
> > - Original Message -
> > >
> > >
> > > On Mon, Sep 19, 2016 at 3:07 PM, Roy Golan < rgo...@redhat.com >
> wrote:
> > >
> > >
> > >
> > >
> > >
> > > On 19 September 2016 at 15:12, Doron Fediuck < dfedi...@redhat.com >
> wrote:
> > >
> > >
> > >
> > > Hi All,
> > >
> > > Martin Sivak has been working on the oVirt engine since June 2013.
> > > His main focus over the years was around MoM, external scheduler,
> > > hosted-engine, and
> > > related engine code.
> > > In the past year Martin took over many of the oVirt scheduler related
> work
> > > and did a lot of improvements there. In total Martin has around 170
> engine
> > > patches already merged.
> > > Within these you will find improving the scheduler's notifications and
> > > error
> > > handling, rewriting the pending resource mechanism which resolved many
> > > issues, VM affinity, affinity labels and multiple improvements for our
> > > scheduling policies and hosted engine.
> > > I would like to thank Martin for his great contribution and hope for
> many
> > > more patches to come :)
> > >
> > > I would also like to propose Martin as an engine maintainer for SLA and
> > > Scheduling.
> > > Thanks, Doron
> > >
> > > +1 Actually +2 :)
> > >
> > > +1
> >
> > +1
> >
>
> +1
>

+1


> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > ___
> > > Devel mailing list
> > > Devel@ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/devel
> > >
> > >
> > > ___
> > > Devel mailing list
> > > Devel@ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/devel
> > >
> > >
> > > ___
> > > Devel mailing list
> > > Devel@ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/devel
> > ___
> > Devel mailing list
> > Devel@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/devel
> >
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Bundled jar files in backend subpackage

[Roman Mohr]

Hi,

On Thu, Aug 18, 2016 at 2:57 PM, Juan Hernández  wrote:

> On 08/18/2016 01:56 PM, Sandro Bonazzola wrote:
> > Hi,
> > looking at $ LC_ALL=C rpm -qlvp
> > http://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/
> fc24/noarch/ovirt-engine-backend-4.1.0-0.0.master.
> 20160817221906.git75736af.fc24.noarch.rpm|grep
> > jar |grep -v ^l |grep common
> >
> > I see:
> >
> > -rw-r--r--1 rootroot77761 Aug 18 00:19
> > /usr/share/ovirt-engine/modules/common/com/netflix/
> config/main/archaius-core.jar
> > -rw-r--r--1 rootroot16442 Aug 18 00:19
> > /usr/share/ovirt-engine/modules/common/com/netflix/
> hystrix/contrib/main/hystrix-metrics-event-stream.jar
> > -rw-r--r--1 rootroot   290223 Aug 18 00:19
> > /usr/share/ovirt-engine/modules/common/com/netflix/
> hystrix/main/hystrix-core.jar
> > -rw-r--r--1 rootroot   738300 Aug 18 00:19
> > /usr/share/ovirt-engine/modules/common/io/reactivex/
> rxjava/main/rxjava.jar
> > -rw-r--r--1 rootroot33218 Aug 18 00:19
>
> I think these ^ are all related to the use of Hystrix, Roman will know.
>
>
They are on epel7 and fc23 and newer.

The package names are rxjava, archaius-core, hystrix-metrics-event-stream
and hystrix-core, exactly like the jar name.

For epel7 it will definitely work, on fedora we have to try it out.
There might be a dependency to google guava on the newer archaius. Newest
and very old archaius versions don't have this guava dependency.
At least everything except archaius-core can be taken from fedoa too.


> > /usr/share/ovirt-engine/modules/common/org/apache/
> avalon/framework/main/avalon-framework-api.jar
> > -rw-r--r--1 rootroot61021 Aug 18 00:19
> > /usr/share/ovirt-engine/modules/common/org/apache/
> avalon/framework/main/avalon-framework-impl.jar
> > -rw-r--r--1 rootroot   401858 Aug 18 00:19
> > /usr/share/ovirt-engine/modules/common/org/apache/
> xmlgraphics/batik/main/batik-awt-util.jar
> > -rw-r--r--1 rootroot   558892 Aug 18 00:19
> > /usr/share/ovirt-engine/modules/common/org/apache/
> xmlgraphics/batik/main/batik-bridge.jar
> > -rw-r--r--1 rootroot   310919 Aug 18 00:19
> > /usr/share/ovirt-engine/modules/common/org/apache/
> xmlgraphics/batik/main/batik-css.jar
> > -rw-r--r--1 rootroot10257 Aug 18 00:19
> > /usr/share/ovirt-engine/modules/common/org/apache/
> xmlgraphics/batik/main/batik-ext.jar
> > -rw-r--r--1 rootroot67900 Aug 18 00:19
> > /usr/share/ovirt-engine/modules/common/org/apache/
> xmlgraphics/batik/main/batik-extension.jar
> > -rw-r--r--1 rootroot   242866 Aug 18 00:19
> > /usr/share/ovirt-engine/modules/common/org/apache/
> xmlgraphics/batik/main/batik-gvt.jar
> > -rw-r--r--1 rootroot   601098 Aug 18 00:19
> > /usr/share/ovirt-engine/modules/common/org/apache/
> xmlgraphics/batik/main/batik-svg-dom.jar
> > -rw-r--r--1 rootroot   121997 Aug 18 00:19
> > /usr/share/ovirt-engine/modules/common/org/apache/
> xmlgraphics/batik/main/batik-transcoder.jar
> > -rw-r--r--1 rootroot   128286 Aug 18 00:19
> > /usr/share/ovirt-engine/modules/common/org/apache/
> xmlgraphics/batik/main/batik-util.jar
> > -rw-r--r--1 rootroot   569113 Aug 18 00:19
> > /usr/share/ovirt-engine/modules/common/org/apache/
> xmlgraphics/commons/main/xmlgraphics-commons.jar
> > -rw-r--r--1 rootroot  3079811 Aug 18 00:19
> > /usr/share/ovirt-engine/modules/common/org/apache/
> xmlgraphics/fop/main/fop.jar
>
> Avalon framework, Batik, xmlgraphics and FOP can't be replaced by links,
> because the versions included in Fedora are newer than what we need, and
> they don't work correctly for us.
>
> > -rw-r--r--1 rootroot 6071 Aug 18 00:19
> > /usr/share/ovirt-engine/modules/common/org/ovirt/
> engine/api/metamodel-server/main/metamodel-server.jar
>
> This ^ is an oVirt project, but not distributed as RPM, only via Maven
> Central, so it can't be replaced with a link.
>
> > -rw-r--r--1 rootroot 8225 Aug 18 00:20
> > /usr/share/ovirt-engine/modules/common/org/ovirt/
> engine/core/auth-plugin/main/auth-plugin.jar
> > -rw-r--r--1 rootroot 4012 Aug 18 00:22
> > /usr/share/ovirt-engine/modules/common/org/ovirt/
> engine/core/logger/main/logger.jar
>
> These ^ are part of the engine, they should go in
> /usr/share/java/ovirt-engine, and should be replaced by links.
>
> > -rw-r--r--1 rootroot   370051 Aug 18 00:19
> > /usr/share/ovirt-engine/modules/common/org/springframework/main/spring-
> aop.jar
> > -rw-r--r--1 rootroot   731512 Aug 18 00:19
> > 

[ovirt-devel] Arrays.asList and GWT

[Roman Mohr]

Hi List,

Seems like GWT can't handle Arrays.asList not very well. Sharing this
in case others have similar issues:

I was working on a very strange deserialization bug in combination
with GWT [1]. We had a deserialization problem after loading a VM and
trying to send the VM back to the backend.

The error looked like this:

2016-06-28 12:17:36,206 ERROR [io.undertow.servlet] (default
task-58) Exception while dispatching incoming RPC call:
java.lang.NumberFormatException: For input string: "undefined"

It turned out that GWT could serialize an ArrayList backed by an
Integer[] array but it could not deserialize it afterwards.

In my case a list returned by ResultSet#getArray in the DAO layer was
sent to GWT:

entity.setCpuIds(Arrays.asList((Integer[])
rs.getArray("cpu_core_ids").getArray()));

After copying the content over into a regular ArrayList everything
worked fine again.

entity.setCpuIds(new ArrayList(Arrays.asList((Integer[])
rs.getArray("cpu_core_ids").getArray(

Here is also the gerrit patch set [2].

So if you see strange GWT RPC errors like the one above look out for
"Arrays.asList" in you code :).

Best Regards,

Roman


[1] https://bugzilla.redhat.com/show_bug.cgi?id=1350861
[2] https://gerrit.ovirt.org/#/c/61038/2
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] oVirt metrics

[Roman Mohr]

On Wed, Jul 13, 2016 at 12:36 PM, Yaniv Bronheim  wrote:
> Hi,
>
> In oVirt-4.0 we introduced integration with metrics collectors, In [1] you
> will find a guide for utilizing your environment to retrieve visualized
> reports about hosts and vms statistics.
>
> I encourage to try that out and send us requests for additional valuable
> metrics that you think vdsm should publish.
> This area is still work in progress and we plan to support more technologies
> and different architectures for metrics collections as describes in the
> post. This will follow by additional links in the post ([1]) that describe
> how to do so.. stay tuned.
>
> [1] https://bronhaim.wordpress.com/2016/06/26/ovirt-metrics
>

Very nice work! Looking forward to see the VM stats there :)

Roman

> --
> Yaniv Bronhaim.
>
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Caching of data from the database done properly

[Roman Mohr]

Hi Yevgeny,

On Mon, Jul 11, 2016 at 7:59 PM, Yevgeny Zaspitsky <yzasp...@redhat.com> wrote:
>
>
> On Tue, Jul 5, 2016 at 7:14 AM, Roman Mohr <rm...@redhat.com> wrote:
>>
>> On Mon, Jul 4, 2016 at 11:58 PM, Roman Mohr <rm...@redhat.com> wrote:
>> > Hi Everyone,
>> >
>> > I wanted to discuss a practice which seems to be pretty common in the
>> > engine which I find very limiting, dangerous and for some things it
>> > can even be a blocker.
>> >
>> > There are several places in the engine where we are using maps as
>> > cache in singletons to avoid reloading data from the database. Two
>> > prominent ones are the QuotaManager[1] and the MacPoolPerCluster[2].
>> >
>> > While it looks tempting to just use a map as cache, add some locks
>> > around it and create an injectable singleton, this has some drawbacks:
>> >
>> > 1) We have an autoritative source for our data and it offers
>> > transactions to take care of inconsistencies or parallel updates.
>> > Doing all that in a service again duplicates this.
>> > 2) Caching on the service layer is definitely not a good idea. It can
>> > introduce unwanted side effects when someone invokes the DAOs
>> > directly.
>> > 3) The point is more about the question if a cache is really needed:
>> > Do I just want that cache because I find it convenient to do a
>> > #getMacPoolForCluster(Guid clusterId) in a loop instead of just
>> > loading it once before the loop, or do my usage requirements really
>> > force me to use a cache?
>> >
>> > If you really need a cache, consider the following:
>> >
>> > 1) Do the caching on the DAO layer. This guarantees the best
>> > consistency across the data.
>> > 2) Yes this means either locking in the DAOs or a transactional cache.
>> > But before you complain, think about what in [1] and [2] is done. We
>> > do exactly that there, so the complexity is already introduced anyway.
>> > 3) Since we are working with transactions, a custom cache should NEVER
>> > cache writes (really just talking about our use case here). This makes
>> > checks for existing IDs before adding an entity or similar checks
>> > unnecessary, don't duplicate constraint checks like in [2].
>> > 4) There should always be a way to disable the cache (even if it is
>> > just for testing).
>> > 5) If I can't convince you to move the cache to the DAO layer, still
>> > add a way to disable the cache.
>> >
>>
>> I forgot to mention one thing: There are of course cases where
>> something is loaded on startup. Mostly things which can have multiple
>> sources.
>> For instance for the application configuration itself it is pretty
>> common, or like in the scheduler the scheduling policies where some
>> are Java only,
>> some are coming from other sources. It is still good
>>
>> But for normal business entities accessing parts of it through
>> services and parts of it through services is not the best thing to do
>> (if constructiong the whole business entity out of multiple daos is
>> complex, Repositories can help, but the cache should still be in the
>> dao layer).
>
>
> I do not agree that the caching should be on the DAO layer - that might lead
> to getting an entity that is built of parts that are not coherent each with
> another if the different DAO caches are not in sync.

I can't agree here.
That is what transactions are for. A second layer cache normally
follows transactions. You have interceptors to detect rollbacks and
commits.
If you don't have JTA in place there is then normally a small window
where you can read stale data in different transactions (which is fine
in most cases). It does have nothing to do with where the cache is.

It is much easier to stay in sync since there is no way to by-bass the cache.

> I'd put the cache on the Repositories (non-existent currently) or a higher
> layer, just above the transaction boundaries, so the cache would contain
> service call results rather than raw data.

What does that mean above the Transaction boundaries?
Yes a second level cache is to have a cache across transaction
boundaries and you also have that when you place them in the DAO
layer.

You would further make it very hard to track weather you are allowed
to manipulate data through DAOs, Repositories or Services when you
don't place the basic cache inside the DAOs since you might always by
accident by-pass the cache.

For higher layer caches in singletons it is also almost a prerequisite
to have the basic cache in the DAO layer because you can then also

Re: [ovirt-devel] Caching of data from the database done properly

[Roman Mohr]

> for example: since pool is not a simple cache of db structure, and does not 
> have corresponding data in db layer, it cannot cache writes and it even does 
> not do any writes at all...

Not caching writes yourself is just a general thing. Regarding the
MacPoolPerCluster I was referring to the constraints checks. But again
most of it does not really apply now to that class anyway.

>
> ——
> Surely I can imagine better implementation of this, but it'd require bigger 
> changes whose benefits aren't worth of cost of this change. (I hope that) we 
> have to deal with it. This was original design, and since testing framework 
> (with changed caches or without) should deal with singletons etc. anyways, 
> it's not worthy to change it. If there aren't any better options(I don't 
> know), reinitializing bean can be used (and I apologize for blocking that). 
> I'd avoid bigger rewrites in this area.

For the context: [1]

Hoping for a continuation of the discussion :)

Roman

>
> M.
>
>
> - Original Message -
>> On Mon, Jul 4, 2016 at 11:58 PM, Roman Mohr <rm...@redhat.com> wrote:
>> > Hi Everyone,
>> >
>> > I wanted to discuss a practice which seems to be pretty common in the
>> > engine which I find very limiting, dangerous and for some things it
>> > can even be a blocker.
>> >
>> > There are several places in the engine where we are using maps as
>> > cache in singletons to avoid reloading data from the database. Two
>> > prominent ones are the QuotaManager[1] and the MacPoolPerCluster[2].
>> >
>> > While it looks tempting to just use a map as cache, add some locks
>> > around it and create an injectable singleton, this has some drawbacks:
>> >
>> > 1) We have an autoritative source for our data and it offers
>> > transactions to take care of inconsistencies or parallel updates.
>> > Doing all that in a service again duplicates this.
>> > 2) Caching on the service layer is definitely not a good idea. It can
>> > introduce unwanted side effects when someone invokes the DAOs
>> > directly.
>> > 3) The point is more about the question if a cache is really needed:
>> > Do I just want that cache because I find it convenient to do a
>> > #getMacPoolForCluster(Guid clusterId) in a loop instead of just
>> > loading it once before the loop, or do my usage requirements really
>> > force me to use a cache?
>> >
>> > If you really need a cache, consider the following:
>> >
>> > 1) Do the caching on the DAO layer. This guarantees the best
>> > consistency across the data.
>> > 2) Yes this means either locking in the DAOs or a transactional cache.
>> > But before you complain, think about what in [1] and [2] is done. We
>> > do exactly that there, so the complexity is already introduced anyway.
>> > 3) Since we are working with transactions, a custom cache should NEVER
>> > cache writes (really just talking about our use case here). This makes
>> > checks for existing IDs before adding an entity or similar checks
>> > unnecessary, don't duplicate constraint checks like in [2].
>> > 4) There should always be a way to disable the cache (even if it is
>> > just for testing).
>> > 5) If I can't convince you to move the cache to the DAO layer, still
>> > add a way to disable the cache.
>> >
>>
>> I forgot to mention one thing: There are of course cases where
>> something is loaded on startup. Mostly things which can have multiple
>> sources.
>> For instance for the application configuration itself it is pretty
>> common, or like in the scheduler the scheduling policies where some
>> are Java only,
>> some are coming from other sources. It is still good
>>
>> But for normal business entities accessing parts of it through
>> services and parts of it through services is not the best thing to do
>> (if constructiong the whole business entity out of multiple daos is
>> complex, Repositories can help, but the cache should still be in the
>> dao layer).
>> I hope  you get what I mean.
>>
>> > For as long as there is no general caching solution with something
>> > like ehcache or infinispan, in my eyes such small things matter a lot
>> > to keep a project maintainable.
>> >
>> > That are some of the best practices I have seen around caching
>> > database data. It would be great if we could agree on something like
>> > that. Maybe there is already an agreement on something and I am just
>> > not aware of it.
>> >
>> > Looking forward to hear your feedback.
>> >
>> > Roman
>> >
>> > [1]
>> > https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/quota/QuotaManager.java
>> > [2]
>> > https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/network/macpool/MacPoolPerCluster.java
>> ___
>> Devel mailing list
>> Devel@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>

[1] https://gerrit.ovirt.org/#/c/59912/
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Caching of data from the database done properly

[Roman Mohr]

On Mon, Jul 4, 2016 at 11:58 PM, Roman Mohr <rm...@redhat.com> wrote:
> Hi Everyone,
>
> I wanted to discuss a practice which seems to be pretty common in the
> engine which I find very limiting, dangerous and for some things it
> can even be a blocker.
>
> There are several places in the engine where we are using maps as
> cache in singletons to avoid reloading data from the database. Two
> prominent ones are the QuotaManager[1] and the MacPoolPerCluster[2].
>
> While it looks tempting to just use a map as cache, add some locks
> around it and create an injectable singleton, this has some drawbacks:
>
> 1) We have an autoritative source for our data and it offers
> transactions to take care of inconsistencies or parallel updates.
> Doing all that in a service again duplicates this.
> 2) Caching on the service layer is definitely not a good idea. It can
> introduce unwanted side effects when someone invokes the DAOs
> directly.
> 3) The point is more about the question if a cache is really needed:
> Do I just want that cache because I find it convenient to do a
> #getMacPoolForCluster(Guid clusterId) in a loop instead of just
> loading it once before the loop, or do my usage requirements really
> force me to use a cache?
>
> If you really need a cache, consider the following:
>
> 1) Do the caching on the DAO layer. This guarantees the best
> consistency across the data.
> 2) Yes this means either locking in the DAOs or a transactional cache.
> But before you complain, think about what in [1] and [2] is done. We
> do exactly that there, so the complexity is already introduced anyway.
> 3) Since we are working with transactions, a custom cache should NEVER
> cache writes (really just talking about our use case here). This makes
> checks for existing IDs before adding an entity or similar checks
> unnecessary, don't duplicate constraint checks like in [2].
> 4) There should always be a way to disable the cache (even if it is
> just for testing).
> 5) If I can't convince you to move the cache to the DAO layer, still
> add a way to disable the cache.
>

I forgot to mention one thing: There are of course cases where
something is loaded on startup. Mostly things which can have multiple
sources.
For instance for the application configuration itself it is pretty
common, or like in the scheduler the scheduling policies where some
are Java only,
some are coming from other sources. It is still good

But for normal business entities accessing parts of it through
services and parts of it through services is not the best thing to do
(if constructiong the whole business entity out of multiple daos is
complex, Repositories can help, but the cache should still be in the
dao layer).
I hope  you get what I mean.

> For as long as there is no general caching solution with something
> like ehcache or infinispan, in my eyes such small things matter a lot
> to keep a project maintainable.
>
> That are some of the best practices I have seen around caching
> database data. It would be great if we could agree on something like
> that. Maybe there is already an agreement on something and I am just
> not aware of it.
>
> Looking forward to hear your feedback.
>
> Roman
>
> [1] 
> https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/quota/QuotaManager.java
> [2] 
> https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/network/macpool/MacPoolPerCluster.java
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

[ovirt-devel] Caching of data from the database done properly

[Roman Mohr]

Hi Everyone,

I wanted to discuss a practice which seems to be pretty common in the
engine which I find very limiting, dangerous and for some things it
can even be a blocker.

There are several places in the engine where we are using maps as
cache in singletons to avoid reloading data from the database. Two
prominent ones are the QuotaManager[1] and the MacPoolPerCluster[2].

While it looks tempting to just use a map as cache, add some locks
around it and create an injectable singleton, this has some drawbacks:

1) We have an autoritative source for our data and it offers
transactions to take care of inconsistencies or parallel updates.
Doing all that in a service again duplicates this.
2) Caching on the service layer is definitely not a good idea. It can
introduce unwanted side effects when someone invokes the DAOs
directly.
3) The point is more about the question if a cache is really needed:
Do I just want that cache because I find it convenient to do a
#getMacPoolForCluster(Guid clusterId) in a loop instead of just
loading it once before the loop, or do my usage requirements really
force me to use a cache?

If you really need a cache, consider the following:

1) Do the caching on the DAO layer. This guarantees the best
consistency across the data.
2) Yes this means either locking in the DAOs or a transactional cache.
But before you complain, think about what in [1] and [2] is done. We
do exactly that there, so the complexity is already introduced anyway.
3) Since we are working with transactions, a custom cache should NEVER
cache writes (really just talking about our use case here). This makes
checks for existing IDs before adding an entity or similar checks
unnecessary, don't duplicate constraint checks like in [2].
4) There should always be a way to disable the cache (even if it is
just for testing).
5) If I can't convince you to move the cache to the DAO layer, still
add a way to disable the cache.

For as long as there is no general caching solution with something
like ehcache or infinispan, in my eyes such small things matter a lot
to keep a project maintainable.

That are some of the best practices I have seen around caching
database data. It would be great if we could agree on something like
that. Maybe there is already an agreement on something and I am just
not aware of it.

Looking forward to hear your feedback.

Roman

[1] 
https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/quota/QuotaManager.java
[2] 
https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/network/macpool/MacPoolPerCluster.java
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Integration tests future (and very nice alternative for the DAO fixture file)

[Roman Mohr]

Hi,

an update on the ovirt-engine application wide tests.
When [6] is merged, everything is ready to use.

Please have a look on the post here:


http://rmohr.github.io/continuous%20integration/2016/07/04/application-scoped-tests-for-ovirt-with-arquillian

for a How-To and why it is implemented the way it is.

Reply here or ping me anytime if you have questions or want some help.

Happy testing :)
Roman

On Wed, Apr 13, 2016 at 3:22 PM, Roman Mohr <rm...@redhat.com> wrote:
> Hi all,
>
> In [1] you can find some patches which are meant to improve the test writing
> experience in ovirt-engine.
>
> They provide the following things:
>
>   A) Domain Object builders which can be used for creating and/or persisting
> domain objects [2]
>   B) DAO testing without writing fixtures because of the builders
>   C) Integration testing for commands in conjunction with a real database
> Arquillian, injectable commands and the builders [3]
>
> # How to run what?
>
> A) In normal unit tests just create a new instance of a builder and use it.
> This should help us to get rid of all the small createDefaultVm(),
> createHostWithX() helper methods in our tests.
>
> B) In dao tests just inject them and go ahead. The advantage of not using
> the fixture file is that we can now set up clean scenarios for every test in
> a setup method. See example 2 below on how easy it is to set up a new
> cluster.
>
> C) Arquillian integration tests need to be marked with
> "@Category(IntegrationTest.class)" and can inherit from
> TransactionalTestBase. The @Category annotation makes sure that the
> integration tests are only run when
>
> mvn clean verify -DskipITs=false
>
> is invoked. Note that these tests are then executed in the integration test
> phase of maven. For them we use the maven-failsafe-plugin[5] which will also
> make sure that the testing database is up to date. See [4] for more details.
>
> # Examples
>
> 1) Add a running VM to a host, persist everything to the database and load
> all VMs which are running on the host:
>
> VDS host = vdsBuilder.cluster(persistedCluster).persist();
> vmBuilder.host(host).up().persist();
> List vms = vmDao.getAllRunningForVds(host.getId());
>
> 2) Add 10 hosts with 1 GB of RAM to a cluster, persist the hosts to the
> database in a DAO test:
>
> public class MyHostDaoTest extends BaseDaoTestCase {
>
> @Inject
> private VdsBuilder vdsBuilder;
>
> @Test
> public void createHosts() {
> VdsBuilder builder =
> vdsBuilder.cluster(persistedCluster).physicalMemory(1000);
> for (int x =0; x < 10; x++){
> builder.id(Guid.newGuid()).persist();
> }
> }
> }
>
> 3) Full integration test with arquillian and the database
>
> @Category(IntegrationTest.class)
> public class VmDaoIntegrationTest extends TransactionalTestBase {
>
> @Inject
> VmDao vmDao;
>
> private final Guid VM1_GUID =
> Guid.createGuidFromString("0fe4bc81-5999-4ab6-80f8-7a4a2d4bfacd");
>
> @Deployment
> public static JavaArchive deploy(){
> return createDeployment();
> }
>
> @Test
> public void shouldFailOnExistingEntity() {
>
> vmBuilder.id(VM1_GUID).cluster(clusterBuilder.reset().persist()).persist();
> // This uses assertThat from assertj:
> assertThat(vmDao.get(VM1_GUID)).isNotNull();
> }
> }
>
> 4) Using the builders in a normal unit test without a database:
>
> VM vm = new VmBuilder().id(Guid.newGuid()).up().build();
>
>
> # How to add your own Domain objects?
>
> There are just a few simple rules:
>
> 1) Your builder should extend org.ovirt.engine.core.builder.AbstractBuilder
>
> 2) Make sure that you only access DAOs injected into the builder during
> #prePersist() and #persist(). This allows to use the #build() method also
> without injections
>
> 3) #prePersist() should set all fields which are necessary to suffice
> database constraints. The fields should only be set if they are not already
> set before by the builder. When following this rule we can always persist
> new objects to the database by simply calling myBuilder.reset().persist().
>
> 4) Mark your builder with @Repository to make them useable for our Spring
> DAO tests and our Arquillian integration tests.
>
> So have a look at the patches at [1] and let me know what you think about
> them.
>
> Best Regards and happy testing,
>
> Roman
>
> [1] https://gerrit.ovirt.org/#/q/topic:integration
> [2] https://gerrit.ovirt.org/#/c/47008/17
> [3] https://gerrit.ovirt.org/#/c/47007/10
> [4] https://gerrit.ovirt.org/#/c/47008/17
> [5] https://maven.apache.org/surefire/maven-failsafe-plugin/

[6] https://gerrit.ovirt.org/#/c/59912/4
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

[ovirt-devel] Failing DAO tests

[Roman Mohr]

Hi we have failing DAO tests becaus of:

https://gerrit.ovirt.org/#/c/56192/

Seems like the cluster can't be saved anymore:

testSave(org.ovirt.engine.core.dao.ClusterDaoTest)  Time elapsed: 9
sec  <<< ERROR!
java.lang.NullPointerException
at 
org.ovirt.engine.core.dao.ClusterDaoImpl.getClusterParamSource(ClusterDaoImpl.java:255)
at org.ovirt.engine.core.dao.ClusterDaoImpl.save(ClusterDaoImpl.java:136)
at 
org.ovirt.engine.core.dao.ClusterDaoTest.testSave(ClusterDaoTest.java:318)

ClusterDaoImpl.java:255 is

.addValue("switch_type",
cluster.getRequiredSwitchTypeForCluster().getOptionValue());

Seems like the the requiredSwitchType is not set for all tests and
when trying to access the getOptionValue() we have a nullpointer
exception.

Fails on my local machine and on travis:
https://s3.amazonaws.com/archive.travis-ci.org/jobs/136184841/log.txt

Best Regards,
Roman
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] GWT developer mode not working on fc23?

[Roman Mohr]

On Tue, Jun 7, 2016 at 2:00 PM, Roman Mohr <rm...@redhat.com> wrote:
> Hi,
>
> I wanted to debug some GWT code the first time on fc23. No matter if I
> try to debug 4.0 or 3.6 I always see in my browser and in the gwt
> developer console:
>
> 00:01:36.758 [ERROR] Unable to load module entry point class
> com.gwtplatform.mvp.client.ApplicationControllerImpl (see associated
> exception for details)
>
> com.google.gwt.core.client.JavaScriptException: (NS_ERROR_FAILURE)
> @com.google.gwt.storage.client.StorageImpl::getItem(Ljava/lang/String;Ljava/lang/String;)([string:
> 'localStorage', string: 'ENGINE_WebAdmin_LogHead']): Component
> returned failure code: 0x80004005 (NS_ERROR_FAILURE)
> [nsIDOMStorage.getItem]
> at 
> com.google.gwt.dev.shell.BrowserChannelServer.invokeJavascript(BrowserChannelServer.java:249)
> at 
> com.google.gwt.dev.shell.ModuleSpaceOOPHM.doInvoke(ModuleSpaceOOPHM.java:136)
> at com.google.gwt.dev.shell.ModuleSpace.invokeNative(ModuleSpace.java:576)
> at 
> com.google.gwt.dev.shell.ModuleSpace.invokeNativeObject(ModuleSpace.java:284)
> at 
> com.google.gwt.dev.shell.JavaScriptHost.invokeNativeObject(JavaScriptHost.java:91)
> at com.google.gwt.storage.client.StorageImpl.getItem(StorageImpl.java)
> at com.google.gwt.storage.client.Storage.getItem(Storage.java:257)
> at 
> org.ovirt.engine.ui.common.system.ClientStorageImpl.getLocalItemImpl(ClientStorageImpl.java:53)
> at 
> org.ovirt.engine.ui.common.system.ClientStorageImpl.getLocalItem(ClientStorageImpl.java:46)
> at 
> org.ovirt.engine.ui.common.logging.LocalStorageLogHandler.readInt(LocalStorageLogHandler.java:56)
> at 
> org.ovirt.engine.ui.common.logging.LocalStorageLogHandler.setActive(LocalStorageLogHandler.java:48)
> at 
> org.ovirt.engine.ui.common.logging.LocalStorageLogHandler.init(LocalStorageLogHandler.java:40)
> at 
> org.ovirt.engine.ui.common.system.BaseApplicationInit.initLocalStorageLogHandler(BaseApplicationInit.java:120)
> at 
> org.ovirt.engine.ui.common.system.BaseApplicationInit.onBootstrap(BaseApplicationInit.java:102)
> at 
> com.gwtplatform.mvp.client.ApplicationControllerImpl.init(ApplicationControllerImpl.java:11)
> at 
> com.gwtplatform.mvp.client.ApplicationControllerImpl.onModuleLoad(ApplicationControllerImpl.java:15)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at com.google.gwt.dev.shell.ModuleSpace.onLoad(ModuleSpace.java:411)
> at 
> com.google.gwt.dev.shell.OophmSessionHandler.loadModule(OophmSessionHandler.java:200)
> at 
> com.google.gwt.dev.shell.BrowserChannelServer.processConnection(BrowserChannelServer.java:526)
> at 
> com.google.gwt.dev.shell.BrowserChannelServer.run(BrowserChannelServer.java:364)
> at java.lang.Thread.run(Thread.java:745)
>

And I am using Firefox 17 for it.

> Thanks,
> Roman
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

[ovirt-devel] GWT developer mode not working on fc23?

[Roman Mohr]

Hi,

I wanted to debug some GWT code the first time on fc23. No matter if I
try to debug 4.0 or 3.6 I always see in my browser and in the gwt
developer console:

00:01:36.758 [ERROR] Unable to load module entry point class
com.gwtplatform.mvp.client.ApplicationControllerImpl (see associated
exception for details)

com.google.gwt.core.client.JavaScriptException: (NS_ERROR_FAILURE)
@com.google.gwt.storage.client.StorageImpl::getItem(Ljava/lang/String;Ljava/lang/String;)([string:
'localStorage', string: 'ENGINE_WebAdmin_LogHead']): Component
returned failure code: 0x80004005 (NS_ERROR_FAILURE)
[nsIDOMStorage.getItem]
at 
com.google.gwt.dev.shell.BrowserChannelServer.invokeJavascript(BrowserChannelServer.java:249)
at com.google.gwt.dev.shell.ModuleSpaceOOPHM.doInvoke(ModuleSpaceOOPHM.java:136)
at com.google.gwt.dev.shell.ModuleSpace.invokeNative(ModuleSpace.java:576)
at com.google.gwt.dev.shell.ModuleSpace.invokeNativeObject(ModuleSpace.java:284)
at 
com.google.gwt.dev.shell.JavaScriptHost.invokeNativeObject(JavaScriptHost.java:91)
at com.google.gwt.storage.client.StorageImpl.getItem(StorageImpl.java)
at com.google.gwt.storage.client.Storage.getItem(Storage.java:257)
at 
org.ovirt.engine.ui.common.system.ClientStorageImpl.getLocalItemImpl(ClientStorageImpl.java:53)
at 
org.ovirt.engine.ui.common.system.ClientStorageImpl.getLocalItem(ClientStorageImpl.java:46)
at 
org.ovirt.engine.ui.common.logging.LocalStorageLogHandler.readInt(LocalStorageLogHandler.java:56)
at 
org.ovirt.engine.ui.common.logging.LocalStorageLogHandler.setActive(LocalStorageLogHandler.java:48)
at 
org.ovirt.engine.ui.common.logging.LocalStorageLogHandler.init(LocalStorageLogHandler.java:40)
at 
org.ovirt.engine.ui.common.system.BaseApplicationInit.initLocalStorageLogHandler(BaseApplicationInit.java:120)
at 
org.ovirt.engine.ui.common.system.BaseApplicationInit.onBootstrap(BaseApplicationInit.java:102)
at 
com.gwtplatform.mvp.client.ApplicationControllerImpl.init(ApplicationControllerImpl.java:11)
at 
com.gwtplatform.mvp.client.ApplicationControllerImpl.onModuleLoad(ApplicationControllerImpl.java:15)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.gwt.dev.shell.ModuleSpace.onLoad(ModuleSpace.java:411)
at 
com.google.gwt.dev.shell.OophmSessionHandler.loadModule(OophmSessionHandler.java:200)
at 
com.google.gwt.dev.shell.BrowserChannelServer.processConnection(BrowserChannelServer.java:526)
at 
com.google.gwt.dev.shell.BrowserChannelServer.run(BrowserChannelServer.java:364)
at java.lang.Thread.run(Thread.java:745)

Thanks,
Roman
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Failing MacPoolDaoTests on master

[Roman Mohr]

On Tue, May 31, 2016 at 11:21 AM, Eyal Edri <ee...@redhat.com> wrote:
> they only run on changes in :
>
> **/backend/manager/modules/common/src/main/java/org/ovirt/engine/core/common/businessentities/**
> **sql
> **/backend/manager/modules/dal/**
>

So where are the jobs for for https://gerrit.ovirt.org/#/c/58328/ for
instance? They should have been triggered, right?

> excluding file
> **/backend/manager/modules/dal/src/main/resources/bundles/**
>
> On Tue, May 31, 2016 at 12:15 PM, Roman Mohr <rm...@redhat.com> wrote:
>>
>> On Tue, May 31, 2016 at 11:12 AM, Eyal Edri <ee...@redhat.com> wrote:
>> > the dao tests 4.0 are on the new jenkins and they are running:
>> >
>> > http://jenkins.ovirt.org/job/ovirt-engine_4.0_dao-unit-tests_merged/
>> > http://jenkins.ovirt.org/job/ovirt-engine_4.0_dao-unit-tests_created/
>> >
>>
>> Am I missing somehting here? From my perspective it looks like they
>> are never triggered.
>>
>> > the old one in old jenkins should be deleted.
>> >
>> > On Tue, May 31, 2016 at 11:57 AM, Roman Mohr <rm...@redhat.com> wrote:
>> >>
>> >> On Tue, May 31, 2016 at 10:49 AM, Roman Mohr <rm...@redhat.com> wrote:
>> >> > On Tue, May 31, 2016 at 10:35 AM, Barak Korren <bkor...@redhat.com>
>> >> > wrote:
>> >> >>> That seems to be another issue. When building locally and on travis
>> >> >>> we
>> >> >>> have failing dao tests, not missing libraries.
>> >> >>>
>> >> >>> Further, on jenkins it seems like we are not running the DAO tests
>> >> >>> since 26th of May.
>> >> >>> Wrote a mail to infra list regarding that about an hour ago.
>> >> >>
>> >> >> Those are still running on the old Jenkins, we only created them on
>> >> >> the new one for the new 4.0 branch.
>> >> >> http://jenkins-old.ovirt.org/search/?q=dao
>> >> >
>> >> > Ok, the failing tests on merged master patches are there.
>> >> > But we are not running tests on gerrit patches currently:
>> >> >
>> >> >
>> >> > http://jenkins-old.ovirt.org/job/ovirt-engine_4.0_dao-unit-tests_merged/
>> >> >
>> >> > Were they disabled deliberately?
>> >> >
>> >>
>> >> Oh that was the wrong job. So they are enabled:
>> >>
>> >>
>> >>
>> >> http://jenkins-old.ovirt.org/job/ovirt-engine_master_dao-unit-tests_created/
>> >>
>> >> So maybe the issue is that the CI tests from old and new jenkins are
>> >> overwriting their pluses and minuses?
>> >>
>> >> >>
>> >> >> We are working on moving these tests into check_patch.sh so that we
>> >> >> can neutralize environmental influences and stop having to give
>> >> >> theses
>> >> >> tests special treatment.
>> >> >>
>> >> >> --
>> >> >> Barak Korren
>> >> >> bkor...@redhat.com
>> >> >> RHEV-CI Team
>> >> ___
>> >> Devel mailing list
>> >> Devel@ovirt.org
>> >> http://lists.ovirt.org/mailman/listinfo/devel
>> >>
>> >>
>> >
>> >
>> >
>> > --
>> > Eyal Edri
>> > Associate Manager
>> > RHEV DevOps
>> > EMEA ENG Virtualization R
>> > Red Hat Israel
>> >
>> > phone: +972-9-7692018
>> > irc: eedri (on #tlv #rhev-dev #rhev-integ)
>
>
>
>
> --
> Eyal Edri
> Associate Manager
> RHEV DevOps
> EMEA ENG Virtualization R
> Red Hat Israel
>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Failing MacPoolDaoTests on master

[Roman Mohr]

On Tue, May 31, 2016 at 11:12 AM, Eyal Edri <ee...@redhat.com> wrote:
> the dao tests 4.0 are on the new jenkins and they are running:
>
> http://jenkins.ovirt.org/job/ovirt-engine_4.0_dao-unit-tests_merged/
> http://jenkins.ovirt.org/job/ovirt-engine_4.0_dao-unit-tests_created/
>

Am I missing somehting here? From my perspective it looks like they
are never triggered.

> the old one in old jenkins should be deleted.
>
> On Tue, May 31, 2016 at 11:57 AM, Roman Mohr <rm...@redhat.com> wrote:
>>
>> On Tue, May 31, 2016 at 10:49 AM, Roman Mohr <rm...@redhat.com> wrote:
>> > On Tue, May 31, 2016 at 10:35 AM, Barak Korren <bkor...@redhat.com>
>> > wrote:
>> >>> That seems to be another issue. When building locally and on travis we
>> >>> have failing dao tests, not missing libraries.
>> >>>
>> >>> Further, on jenkins it seems like we are not running the DAO tests
>> >>> since 26th of May.
>> >>> Wrote a mail to infra list regarding that about an hour ago.
>> >>
>> >> Those are still running on the old Jenkins, we only created them on
>> >> the new one for the new 4.0 branch.
>> >> http://jenkins-old.ovirt.org/search/?q=dao
>> >
>> > Ok, the failing tests on merged master patches are there.
>> > But we are not running tests on gerrit patches currently:
>> >
>> > http://jenkins-old.ovirt.org/job/ovirt-engine_4.0_dao-unit-tests_merged/
>> >
>> > Were they disabled deliberately?
>> >
>>
>> Oh that was the wrong job. So they are enabled:
>>
>>
>> http://jenkins-old.ovirt.org/job/ovirt-engine_master_dao-unit-tests_created/
>>
>> So maybe the issue is that the CI tests from old and new jenkins are
>> overwriting their pluses and minuses?
>>
>> >>
>> >> We are working on moving these tests into check_patch.sh so that we
>> >> can neutralize environmental influences and stop having to give theses
>> >> tests special treatment.
>> >>
>> >> --
>> >> Barak Korren
>> >> bkor...@redhat.com
>> >> RHEV-CI Team
>> ___
>> Devel mailing list
>> Devel@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>>
>
>
>
> --
> Eyal Edri
> Associate Manager
> RHEV DevOps
> EMEA ENG Virtualization R
> Red Hat Israel
>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Failing MacPoolDaoTests on master

[Roman Mohr]

On Tue, May 31, 2016 at 10:49 AM, Roman Mohr <rm...@redhat.com> wrote:
> On Tue, May 31, 2016 at 10:35 AM, Barak Korren <bkor...@redhat.com> wrote:
>>> That seems to be another issue. When building locally and on travis we
>>> have failing dao tests, not missing libraries.
>>>
>>> Further, on jenkins it seems like we are not running the DAO tests
>>> since 26th of May.
>>> Wrote a mail to infra list regarding that about an hour ago.
>>
>> Those are still running on the old Jenkins, we only created them on
>> the new one for the new 4.0 branch.
>> http://jenkins-old.ovirt.org/search/?q=dao
>
> Ok, the failing tests on merged master patches are there.
> But we are not running tests on gerrit patches currently:
>
> http://jenkins-old.ovirt.org/job/ovirt-engine_4.0_dao-unit-tests_merged/
>
> Were they disabled deliberately?
>

Oh that was the wrong job. So they are enabled:

http://jenkins-old.ovirt.org/job/ovirt-engine_master_dao-unit-tests_created/

So maybe the issue is that the CI tests from old and new jenkins are
overwriting their pluses and minuses?

>>
>> We are working on moving these tests into check_patch.sh so that we
>> can neutralize environmental influences and stop having to give theses
>> tests special treatment.
>>
>> --
>> Barak Korren
>> bkor...@redhat.com
>> RHEV-CI Team
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Failing MacPoolDaoTests on master

[Roman Mohr]

On Tue, May 31, 2016 at 10:35 AM, Barak Korren  wrote:
>> That seems to be another issue. When building locally and on travis we
>> have failing dao tests, not missing libraries.
>>
>> Further, on jenkins it seems like we are not running the DAO tests
>> since 26th of May.
>> Wrote a mail to infra list regarding that about an hour ago.
>
> Those are still running on the old Jenkins, we only created them on
> the new one for the new 4.0 branch.
> http://jenkins-old.ovirt.org/search/?q=dao

Ok, the failing tests on merged master patches are there.
But we are not running tests on gerrit patches currently:

http://jenkins-old.ovirt.org/job/ovirt-engine_4.0_dao-unit-tests_merged/

Were they disabled deliberately?

>
> We are working on moving these tests into check_patch.sh so that we
> can neutralize environmental influences and stop having to give theses
> tests special treatment.
>
> --
> Barak Korren
> bkor...@redhat.com
> RHEV-CI Team
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Failing MacPoolDaoTests on master

[Roman Mohr]

On Tue, May 31, 2016 at 10:13 AM, Barak Korren <bkor...@redhat.com> wrote:
> On 31 May 2016 at 10:45, Roman Mohr <rm...@redhat.com> wrote:
>> Hi,
>>
>> DAO tests are failing on master in
>>
>> org.ovirt.engine.core.dao.MacPoolDaoTest.
>>
>> Build logs are here: 
>> https://travis-ci.org/oVirt/ovirt-engine/builds/134082394
>>
> Or here:
> http://jenkins.ovirt.org/job/ovirt-engine_4.0_dao-unit-tests_created/3/console
>
> It seems there is a vdsm-jsonrpc-java-client build missing?
>
That seems to be another issue. When building locally and on travis we
have failing dao tests, not missing libraries.

Further, on jenkins it seems like we are not running the DAO tests
since 26th of May.
Wrote a mail to infra list regarding that about an hour ago.

> 07:13:31 [ERROR] Failed to execute goal on project
> common-dependencies: Could not resolve dependencies for project
> org.ovirt.engine.core.manager:common-dependencies:jar:4.0.0: Could not
> find artifact org.ovirt.vdsm-jsonrpc-java:vdsm-jsonrpc-java-client:jar:1.2.3
> in ovirt-maven-repository
> (http://artifactory.ovirt.org/artifactory/ovirt-mirror) -> [Help 1]
> 07:13:31 [ERROR]
>
>
> --
> Barak Korren
> bkor...@redhat.com
> RHEV-CI Team
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

[ovirt-devel] Failing MacPoolDaoTests on master

[Roman Mohr]

Hi,

DAO tests are failing on master in

org.ovirt.engine.core.dao.MacPoolDaoTest.

Build logs are here: https://travis-ci.org/oVirt/ovirt-engine/builds/134082394

Best Regards,
Roman
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Doing a mock ovirt-engine build for epel7 on fedora 23 hangs

[Roman Mohr]

On Tue, May 10, 2016 at 12:07 PM, Eyal Edri <ee...@redhat.com> wrote:
> Adding infra,  i think its a known issue which we hit on ci as well
>
After a reboot the problem disappeared.
Will have a closer look at all logs the next time it happens.

> On May 10, 2016 12:53 PM, "Roman Mohr" <rm...@redhat.com> wrote:
>>
>> Hi,
>>
>> My mock rpm builds hang on the line
>>
>> Downloading:
>>
>> http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-ear-plugin/2.8/maven-ear-plugin-2.8.pom
>>
>> Does anyone else have similar issues when building inside mock for
>> epel7 on a fedora 23 machine?\
>>
>> Best Regards,
>> Roman
>> ___
>> Devel mailing list
>> Devel@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>>
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

[ovirt-devel] Doing a mock ovirt-engine build for epel7 on fedora 23 hangs

[Roman Mohr]

Hi,

My mock rpm builds hang on the line

Downloading:
http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-ear-plugin/2.8/maven-ear-plugin-2.8.pom

Does anyone else have similar issues when building inside mock for
epel7 on a fedora 23 machine?\

Best Regards,
Roman
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [oVirt 4.0 Localization Question #3] "NUMA node pinned index error."

[Roman Mohr]

Hi Yuko,

On Wed, May 4, 2016 at 5:44 AM, Yuko Katabami  wrote:

> Hi oVirt developers,
>
> Here is our question #3.
> Thank you!
> ​​
>
> ​​File: AppErrors
> Resource ID: VM_NUMA_NODE_PINNED_INDEX_ERROR
> String: NUMA node pinned index error.
> Question: We cannot understand this message clearly as there are several
> different ways to interpret it.
> Could anyone please paraphrase this string?
> Does it mean, something like "An error ocurred with index of a pinned NUMA
> node."?
>
>
That is definitely an unlucky description.  This is more accurate:
 "Pinning host numa node to vm numa node requested but host numa node index
is not specified."

A user should actually never see that when we do all validations right. We
validate here numa node mappings in backend commands after REST and
frontend validations.


> Yuko
>
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>

Roman
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Master ovirt-engine build broken

[Roman Mohr]

On Wed, May 4, 2016 at 9:44 AM, Tomas Jelinek <tjeli...@redhat.com> wrote:

> hmmm,
> regression introduced yesterday by https://gerrit.ovirt.org/#/c/56720/
> fix on the way
>
> Thx


> - Original Message -----
> > From: "Roman Mohr" <rm...@redhat.com>
> > To: "devel" <devel@ovirt.org>
> > Sent: Wednesday, May 4, 2016 9:34:53 AM
> > Subject: [ovirt-devel] Master ovirt-engine build broken
> >
> > Hi,
> >
> > I see failing tests regarding to missing translations (e.g. [1]).
> >
> > [...]
> >
> > Failed tests:   doTest(org.ovirt.engine.ui.uicompat.UIMessagesTest):
> > cpuInfoLabel does not match the number of parameters in
> > UIMessages_zh_CN.properties(..)
> >
> > [...]
> >
> > Best Regards,
> >
> > Roman
> >
> > [1]
> >
> http://jenkins.ovirt.org/job/ovirt-engine_master_check-patch-fc23-x86_64/407/console
> >
> > ___
> > Devel mailing list
> > Devel@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/devel
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

[ovirt-devel] Master ovirt-engine build broken

[Roman Mohr]

Hi,

I see failing tests regarding to missing translations (e.g. [1]).

[...]

Failed tests:   doTest(org.ovirt.engine.ui.uicompat.UIMessagesTest):
cpuInfoLabel does not match the number of parameters in
UIMessages_zh_CN.properties(..)


[...]

Best Regards,

Roman

[1]
http://jenkins.ovirt.org/job/ovirt-engine_master_check-patch-fc23-x86_64/407/console
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Integration tests future (and very nice alternative for the DAO fixture file)

[Roman Mohr]

On Sun, Apr 17, 2016 at 9:49 PM, Martin Mucha <mmu...@redhat.com> wrote:

> Having such builders would simplify our lives for sure. But I'd really try
> to avoid any autogeneration. I would cost lots of man hours to make it
> happen and result won't be good. If we agree to use this approach, and
> everyone write new methods as they're needed, 'some basic builders' come to
> life very easy and fast.


That's what I had in mind and I have pretty good experience with that
pattern from previous projects. I am also against autogeneration for the
mentioned reasons in that case. If you are on a greenfield project, Lombok
has a nice @Builder [1] in general but it does also not fit very well here.


> Also some places already has such similar builders — like
> org.ovirt.engine.core.bll.network.host.HostSetupNetworksValidatorTest.HostSetupNetworksValidatorBuilder.
>
> M.
>
> - Original Message -
> > Hi Roman,
> >
> > I really like the idea behind domain object builders.
> >
> > Maybe a silly question, but how much effort would it take
> > to autogenerate some "basic" builders from domain objects?
> > (without DAO/Integration test related stuff like prePersist
> > override etc.)
> >
> > From the "basic" builders we could derive DAO/Integration
> > ones as subclasses. This way, the domain object specific
> > stuff wouldn't have to be kept in sync by hand, and we can
> > subclass only if we need to make a builder DAO/Integration
> > test aware.
> >
> > Vojtech
> >
> >
> > - Original Message -
> > > From: "Roman Mohr" <rm...@redhat.com>
> > > To: "Eyal Edri" <ee...@redhat.com>
> > > Cc: "Juan Antonio Hernandez Fernandez" <jhern...@redhat.com>, "devel"
> > > <devel@ovirt.org>
> > > Sent: Thursday, April 14, 2016 1:05:25 PM
> > > Subject: Re: [ovirt-devel] Integration tests future (and very nice
> > > alternative for the DAO fixture file)
> > >
> > >
> > >
> > > On Thu, Apr 14, 2016 at 12:32 PM, Eyal Edri < ee...@redhat.com >
> wrote:
> > >
> > >
> > >
> > > Will that replace the current DAO tests running in CI?
> > >
> > >
> > > For now no. What you can do with the builders reagarding to the DAO
> tests
> > > is
> > > creating test scenarios for the database. So instead of adding
> entities to
> > > the fixture file you can set up clean sceanrios for your tests just
> with
> > > the
> > > builders in @Test or @Before methods.
> > >
> > > Integration tests are using Arqullian with the spring transaction
> manager
> > > and
> > > will probably an extra CI job which passes the right maven flags.
> > > Arquillian
> > > is nice here because we are much closer to a real JBoss than with
> Spring.
> > >
> > > The reason why we do not use Arquillian and only the builders for the
> DAO
> > > test is that you would need a full JBoss downloaded in the background
> to
> > > give us a transaction manager which does the same thing as springs
> > > transaction manager does during the build.
> > >
> > > The JBoss people are currently working on modularizing all their JBoss
> > > libraries (for JBoss Swarm) and I hope that in the future we can drop
> the
> > > spring transaction manager and do everything with arquillian and the
> JBoss
> > > transaction manager.
> > >
> > >
> > >
> > > On Wed, Apr 13, 2016 at 4:22 PM, Roman Mohr < rm...@redhat.com >
> wrote:
> > >
> > >
> > >
> > > Hi all,
> > >
> > > In [1] you can find some patches which are meant to improve the test
> > > writing
> > > experience in ovirt-engine.
> > >
> > > They provide the following things:
> > >
> > > A) Domain Object builders which can be used for creating and/or
> persisting
> > > domain objects [2]
> > > B) DAO testing without writing fixtures because of the builders
> > > C) Integration testing for commands in conjunction with a real database
> > > Arquillian, injectable commands and the builders [3]
> > >
> > > # How to run what?
> > >
> > > A) In normal unit tests just create a new instance of a builder and
> use it.
> > > This should help us to get rid of all the small createDefaultVm(),
> > > createHostWithX() helper methods in our tests.
> > >
> > > B) In dao

Re: [ovirt-devel] Integration tests future (and very nice alternative for the DAO fixture file)

[Roman Mohr]

On Thu, Apr 14, 2016 at 12:32 PM, Eyal Edri <ee...@redhat.com> wrote:

> Will that replace the current DAO tests running in CI?
>
>
For now no. What you can do with the builders reagarding to the DAO tests
is creating test scenarios for the database. So instead of adding entities
to the fixture file you can set up clean sceanrios for your tests just with
the builders in @Test or @Before methods.

Integration tests are using Arqullian with the spring transaction manager
and will probably an extra CI job which passes the right maven flags.
Arquillian is nice here because we are much closer to a real JBoss than
with Spring.

The reason why we do not use Arquillian and only the builders for the DAO
test is that you would need a full JBoss downloaded in the background to
give us a transaction manager which does the same thing as springs
transaction manager does during the build.

The JBoss people are currently working on modularizing all their JBoss
libraries (for JBoss Swarm) and I hope that in the future we can drop the
spring transaction manager and do everything with arquillian and the JBoss
transaction manager.


> On Wed, Apr 13, 2016 at 4:22 PM, Roman Mohr <rm...@redhat.com> wrote:
>
>> Hi all,
>>
>> In [1] you can find some patches which are meant to improve the test
>> writing experience in ovirt-engine.
>>
>> They provide the following things:
>>
>>   A) Domain Object builders which can be used for creating and/or
>> persisting domain objects [2]
>>   B) DAO testing without writing fixtures because of the builders
>>   C) Integration testing for commands in conjunction with a real database
>> Arquillian, injectable commands and the builders [3]
>>
>> # How to run what?
>>
>> A) In normal unit tests just create a new instance of a builder and use
>> it. This should help us to get rid of all the small createDefaultVm(),
>> createHostWithX() helper methods in our tests.
>>
>> B) In dao tests just inject them and go ahead. The advantage of not using
>> the fixture file is that we can now set up clean scenarios for every test
>> in a setup method. See example 2 below on how easy it is to set up a new
>> cluster.
>>
>> C) Arquillian integration tests need to be marked with
>> "@Category(IntegrationTest.class)" and can inherit from
>> TransactionalTestBase. The @Category annotation makes sure that the
>> integration tests are only run when
>>
>> mvn clean verify -DskipITs=false
>>
>> is invoked. Note that these tests are then executed in the integration
>> test phase of maven. For them we use the maven-failsafe-plugin[5] which
>> will also make sure that the testing database is up to date. See [4] for
>> more details.
>>
>> # Examples
>>
>> 1) Add a running VM to a host, persist everything to the database and
>> load all VMs which are running on the host:
>>
>> VDS host = vdsBuilder.cluster(persistedCluster).persist();
>> vmBuilder.host(host).up().persist();
>> List vms = vmDao.getAllRunningForVds(host.getId());
>>
>> 2) Add 10 hosts with 1 GB of RAM to a cluster, persist the hosts to the
>> database in a DAO test:
>>
>> public class MyHostDaoTest extends BaseDaoTestCase {
>>
>> @Inject
>> private VdsBuilder vdsBuilder;
>>
>> @Test
>> public void createHosts() {
>> VdsBuilder builder =
>> vdsBuilder.cluster(persistedCluster).physicalMemory(1000);
>> for (int x =0; x < 10; x++){
>> builder.id(Guid.newGuid()).persist();
>> }
>> }
>> }
>>
>> 3) Full integration test with arquillian and the database
>>
>> @Category(IntegrationTest.class)
>> public class VmDaoIntegrationTest extends TransactionalTestBase {
>>
>> @Inject
>> VmDao vmDao;
>>
>> private final Guid VM1_GUID =
>> Guid.createGuidFromString("0fe4bc81-5999-4ab6-80f8-7a4a2d4bfacd");
>>
>> @Deployment
>> public static JavaArchive deploy(){
>> return createDeployment();
>> }
>>
>> @Test
>> public void shouldFailOnExistingEntity() {
>>
>> vmBuilder.id(VM1_GUID).cluster(clusterBuilder.reset().persist()).persist();
>> // This uses assertThat from assertj:
>> assertThat(vmDao.get(VM1_GUID)).isNotNull();
>> }
>> }
>>
>> 4) Using the builders in a normal unit test without a database:
>>
>> VM vm = new VmBuilder().id(Guid.newGuid()).up().build();

[ovirt-devel] Integration tests future (and very nice alternative for the DAO fixture file)

[Roman Mohr]

Hi all,

In [1] you can find some patches which are meant to improve the test
writing experience in ovirt-engine.

They provide the following things:

  A) Domain Object builders which can be used for creating and/or
persisting domain objects [2]
  B) DAO testing without writing fixtures because of the builders
  C) Integration testing for commands in conjunction with a real database
Arquillian, injectable commands and the builders [3]

# How to run what?

A) In normal unit tests just create a new instance of a builder and use it.
This should help us to get rid of all the small createDefaultVm(),
createHostWithX() helper methods in our tests.

B) In dao tests just inject them and go ahead. The advantage of not using
the fixture file is that we can now set up clean scenarios for every test
in a setup method. See example 2 below on how easy it is to set up a new
cluster.

C) Arquillian integration tests need to be marked with
"@Category(IntegrationTest.class)" and can inherit from
TransactionalTestBase. The @Category annotation makes sure that the
integration tests are only run when

mvn clean verify -DskipITs=false

is invoked. Note that these tests are then executed in the integration test
phase of maven. For them we use the maven-failsafe-plugin[5] which will
also make sure that the testing database is up to date. See [4] for more
details.

# Examples

1) Add a running VM to a host, persist everything to the database and load
all VMs which are running on the host:

VDS host = vdsBuilder.cluster(persistedCluster).persist();
vmBuilder.host(host).up().persist();
List vms = vmDao.getAllRunningForVds(host.getId());

2) Add 10 hosts with 1 GB of RAM to a cluster, persist the hosts to the
database in a DAO test:

public class MyHostDaoTest extends BaseDaoTestCase {

@Inject
private VdsBuilder vdsBuilder;

@Test
public void createHosts() {
VdsBuilder builder =
vdsBuilder.cluster(persistedCluster).physicalMemory(1000);
for (int x =0; x < 10; x++){
builder.id(Guid.newGuid()).persist();
}
}
}

3) Full integration test with arquillian and the database

@Category(IntegrationTest.class)
public class VmDaoIntegrationTest extends TransactionalTestBase {

@Inject
VmDao vmDao;

private final Guid VM1_GUID =
Guid.createGuidFromString("0fe4bc81-5999-4ab6-80f8-7a4a2d4bfacd");

@Deployment
public static JavaArchive deploy(){
return createDeployment();
}

@Test
public void shouldFailOnExistingEntity() {

vmBuilder.id(VM1_GUID).cluster(clusterBuilder.reset().persist()).persist();
// This uses assertThat from assertj:
assertThat(vmDao.get(VM1_GUID)).isNotNull();
}
}

4) Using the builders in a normal unit test without a database:

VM vm = new VmBuilder().id(Guid.newGuid()).up().build();


# How to add your own Domain objects?

There are just a few simple rules:

1) Your builder should extend org.ovirt.engine.core.builder.AbstractBuilder

2) Make sure that you only access DAOs injected into the builder during
#prePersist() and #persist(). This allows to use the #build() method also
without injections

3) #prePersist() should set all fields which are necessary to suffice
database constraints. The fields should only be set if they are not already
set before by the builder. When following this rule we can always persist
new objects to the database by simply calling myBuilder.reset().persist().

4) Mark your builder with @Repository to make them useable for our Spring
DAO tests and our Arquillian integration tests.

So have a look at the patches at [1] and let me know what you think about
them.

Best Regards and happy testing,

Roman

[1] https://gerrit.ovirt.org/#/q/topic:integration
[2] https://gerrit.ovirt.org/#/c/47008/17
[3] https://gerrit.ovirt.org/#/c/47007/10
[4] https://gerrit.ovirt.org/#/c/47008/17
[5] https://maven.apache.org/surefire/maven-failsafe-plugin/
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] ovirt-vdsmfake changes

[Roman Mohr]

Am 18.02.2016 12:36 schrieb "Michal Skrivanek" <michal.skriva...@redhat.com
>:
>
>
>> On 18 Feb 2016, at 10:44, Roy Golan <rgo...@redhat.com> wrote:
>>
>>
>>
>> On Tue, Feb 16, 2016 at 2:45 PM, Roman Mohr <rm...@redhat.com> wrote:
>>>
>>> Hi,
>>>
>>> A small heads up regarding to ovirt-vdsmfake. We merged some changes
today which add support for oVirt 4.0 and make it much easier to get
started with vdsmfake.
>>>
>>> With the changes, now it is enough to just run `mvn jetty:run` to have
a working vdsmfake.
>>> Have a look at the new quickstart guide in the README.md [1] to see how
to get started..
>>>
>>> Best regards,
>>> Roman
>>>
>>> [1] https://github.com/oVirt/ovirt-vdsmfake
>>
>>
>>
>> Thanks, nice for a real quick start.
>>
>> I can add that I also use dnsmasq to resolve any address to an IP
>>
>>  dnsmasq --address=/vdsm.simulator/127.0.0.1
>>
>> and that will resolve XXX.vdsm.simulator to 127.0.0.1. Just need to add
nameserver to /etc/resolv.conf
>>
>> I'll add that to the README as well
>
>
> please make it the favorite solution. It’s way more clean than messing
with /etc/hosts
>
+1
>>
>> ___
>> Devel mailing list
>> Devel@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] ovirt-vdsmfake changes

[Roman Mohr]

On Thu, Feb 18, 2016 at 10:44 AM, Roy Golan <rgo...@redhat.com> wrote:

>
>
> On Tue, Feb 16, 2016 at 2:45 PM, Roman Mohr <rm...@redhat.com> wrote:
>
>> Hi,
>>
>> A small heads up regarding to ovirt-vdsmfake. We merged some changes
>> today which add support for oVirt 4.0 and make it much easier to get
>> started with vdsmfake.
>>
>> With the changes, now it is enough to just run `mvn jetty:run` to have a
>> working vdsmfake.
>> Have a look at the new quickstart guide in the README.md [1] to see how
>> to get started..
>>
>> Best regards,
>> Roman
>>
>> [1] https://github.com/oVirt/ovirt-vdsmfake
>>
>
>
> Thanks, nice for a real quick start.
>
> I can add that I also use dnsmasq to resolve any address to an IP
>
>  *dnsmasq --address=/vdsm.simulator/127.0.0.1 <http://127.0.0.1>*
>
> and that will resolve *XXX.vdsm.simulator* to 127.0.0.1. Just need to add
> nameserver to /etc/resolv.conf
>
> I'll add that to the README as well
>
>
Very useful. Thanks for sharing!
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

[ovirt-devel] ovirt-vdsmfake changes

[Roman Mohr]

Hi,

A small heads up regarding to ovirt-vdsmfake. We merged some changes today
which add support for oVirt 4.0 and make it much easier to get started with
vdsmfake.

With the changes, now it is enough to just run `mvn jetty:run` to have a
working vdsmfake.
Have a look at the new quickstart guide in the README.md [1] to see how to
get started..

Best regards,
Roman

[1] https://github.com/oVirt/ovirt-vdsmfake
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] 10 Patches in ovirt-engine-3.6 and not in origin/ovirt-engine-3.6.3 branch according to Change-Id

[Roman Mohr]

On Mon, Feb 8, 2016 at 5:29 PM, Sandro Bonazzola <sbona...@redhat.com>
wrote:

> Hi, just an heads up that some patches are on 3.6 branch and not in 3.6.3.
> Some of them have bugs targeted to 3.6.5 so they should be fine but some
> have target 3.6.3, please review.
>
> $ ./diff_branches.py ovirt-engine-3.6 origin/ovirt-engine-3.6.3
> 19276 patches on ovirt-engine-3.6
> 19270 patches on origin/ovirt-engine-3.6.3
> commit f0c0bb798fd5a7dff6a2c42256e453a3c8210015
> Author: Moti Asayag <masa...@redhat.com>
> Date:   Thu Jan 28 11:17:44 2016 +0200
> webadmin: Uninitialized host fails to open hardware tab
> Change-Id: I759e49a0418c9a0c21e917b153d4c1ef4c877911
> Bug-Url: https://bugzilla.redhat.com/1293574
>
> commit 560347fd35cf223d857c00b2a6e4f9f9b5d6bf8d
> Author: Martin Mucha <mmu...@redhat.com>
> Date:   Mon Nov 30 15:53:13 2015 +0100
> core: Do not acquire in left-most-available order
> Change-Id: I00f4ebf8371c1a0e531baf7ef764c99d0be63ab2
> Bug-Url: https://bugzilla.redhat.com/1269301
>
> commit cc2c8521de33b03cb1d8c7cec1d761c8cd7273ee
> Author: Tomas Jelinek <tjeli...@redhat.com>
> Date:   Thu Jan 28 14:04:49 2016 +0100
> webadmin: change sizes of suggest boxes
> Change-Id: Ibdddf508a61c36f01f54b5f6f1eab1a7a46ad92c
> Bug-Url: https://bugzilla.redhat.com/1293154
>
> commit 254f50814633dc08aa9a500619ff1ea667139141
> Author: Tomas Jelinek <tjeli...@redhat.com>
> Date:   Wed Jan 27 17:09:28 2016 +0100
> webadmin: new pool dialog contained the sub version field
> Change-Id: I36188c0e666377ea076caea1966004a47632f338
> Bug-Url: https://bugzilla.redhat.com/1302372
>
> commit bc1479c7b908a00c688ddb2b73bcb0ce90788485
> Author: Jakub Niedermertl <jnied...@redhat.com>
> Date:   Thu Jan 28 14:52:45 2016 +0100
> webadmin: Consistent relation memory - guaranteed memory
> Change-Id: I2c1d257b6edc81964272474510afd004248b9e39
> Bug-Url: https://bugzilla.redhat.com/1302582
>
> commit 728cd7776006a1f55bbe67ed020591158567ade3
> Author: Tomas Jelinek <tjeli...@redhat.com>
> Date:   Thu Jan 28 15:55:13 2016 +0100
> webadmin: show the "latest" template only when the new VM is stateless
> Change-Id: I5b2a7beeb2687521e73168d5171ef14a98185e06
> Bug-Url: https://bugzilla.redhat.com/1293154
>
> commit 2acc6808680d7f8abb9129d1d0b6bd955cb68778
> Author: Roman Mohr <rm...@redhat.com>
> Date:   Tue Dec 22 16:32:47 2015 +0100
> core: Only allow continuous numa node indices
> Change-Id: I05e38768be59d9c2eee8a114a3e6d1f684ab5d2d
> Bug-Url: https://bugzilla.redhat.com/1126180
>
>
^^^ merged to 3.6.3 before the build yesterday ^^^



> commit cd164254694a88b992764dba21b2a229aea66a60
> Author: Shmuel Melamud <smela...@redhat.com>
> Date:   Wed Feb 3 19:03:16 2016 +0200
> core: Case-insensitive match of file name patterns
> Change-Id: I2948e0bc18b86a5eb98647ee9dd4b8e0c572aedb
> Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1154391
>
> commit 268027756f5a87828ecf5da41d38cd58e9a5c0f9
> Author: Juan Hernandez <juan.hernan...@redhat.com>
> Date:   Sun Nov 8 19:50:04 2015 +0100
> restapi: Fix RSDL for adding clusters
> Change-Id: I3418e301679c31eec3950c4428485d48e39cad73
> Bug-Url: https://bugzilla.redhat.com/1279159
>
> commit c4b4fe05352082a18c5e401d25157eb65b64913d
> Author: Martin Mucha <mmu...@redhat.com>
> Date:   Wed Feb 3 14:39:01 2016 +0100
> core: add code setting storage pool id in
> DetachNetworkFromVdsInterfaceCommand
> Change-Id: I2bd3d7bcb8a6bbd667d9bbde6658a8746f195692
> Bug-Url: https://bugzilla.redhat.com/1299630
>
>
> 10 Patches in ovirt-engine-3.6 and not in origin/ovirt-engine-3.6.3 branch
> according to Change-Id
>
> --
> Sandro Bonazzola
> Better technology. Faster innovation. Powered by community collaboration.
> See how it works at redhat.com
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Proposing Martin Betak as oVirt UI maintainer

[Roman Mohr]

On Mon, Feb 8, 2016 at 10:28 AM, Martin Betak  wrote:

> Thanks everyone for their expressed support!
>
> I hope I'll be able to continue working towards improving our existing
> frontend codebase and whatever may serve as the UI in the future :-)
>
> Best regards,
>
> Martin B.
>
>
Congratulations!


> - Original Message -
> > From: "Tomas Jelinek" 
> > To: "Martin" 
> > Cc: "devel" 
> > Sent: Monday, February 8, 2016 10:19:34 AM
> > Subject: Re: [ovirt-devel] Proposing Martin Betak as oVirt UI maintainer
> >
> > congrats! :)
> >
> > - Original Message -
> > > From: "David Caro" 
> > > To: "Vojtech Szocs" 
> > > Cc: "Michal Skrivanek" , "devel"  >
> > > Sent: Monday, February 8, 2016 10:11:30 AM
> > > Subject: Re: [ovirt-devel] Proposing Martin Betak as oVirt UI
> maintainer
> > >
> > > On 02/03 09:00, Vojtech Szocs wrote:
> > > > Hello, UI folks!
> > > >
> > > > I'd like to propose Martin Betak as oVirt UI maintainer.
> > >
> > >
> > > I've added him to the ovirt-engine-webadmin gerrit group, let me know
> if
> > > there's anything else needed
> > >
> > > >
> > > > Over time, Martin made some significant UI contributions:
> > > >
> > > > - improving UiCommon by utilizing Java generics [1]
> > > >
> > > > - adding GinUiBinder [2] to allow @Inject'ing GIN-managed
> > > >   objects into GWT widgets created in ui.xml templates
> > > >
> > > > - upgrade GWT version to 2.6.1 [3] for both oVirt webapps
> > > >
> > > > - providing excellent feedback and ideas on oVirtJS project
> > > >   [4] while demonstrating outstanding JavaScript knowledge
> > > >
> > > > [1] https://gerrit.ovirt.org/#/c/32907/
> > > > [2] https://gerrit.ovirt.org/#/c/34954/
> > > > [3] https://gerrit.ovirt.org/#/c/32135/
> > > > [4] https://gerrit.ovirt.org/#/c/49466/
> > > >
> > > > Personal note: Martin is familiar with oVirt GWT UI infra
> > > > as a whole (GWT-P component model, advanced GWT features,
> > > > UiCommon integration etc). His grasp on JS and its current
> > > > eco-system is very solid.
> > > >
> > > > Regards,
> > > > Vojtech
> > > > ___
> > > > Devel mailing list
> > > > Devel@ovirt.org
> > > > http://lists.ovirt.org/mailman/listinfo/devel
> > >
> > > --
> > > David Caro
> > >
> > > Red Hat S.L.
> > > Continuous Integration Engineer - EMEA ENG Virtualization R
> > >
> > > Tel.: +420 532 294 605
> > > Email: dc...@redhat.com
> > > IRC: dcaro|dcaroest@{freenode|oftc|redhat}
> > > Web: www.redhat.com
> > > RHT Global #: 82-62605
> > >
> > > ___
> > > Devel mailing list
> > > Devel@ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/devel
> >
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [ovirt-users] [QE] oVirt 3.6.1 Hosted Engine test day

[Roman Mohr]

On Mon, Dec 14, 2015 at 8:07 PM, Roy Golan <rgo...@redhat.com> wrote:

>
>
> On Mon, Dec 14, 2015 at 7:07 PM, Roman Mohr <rm...@redhat.com> wrote:
>
>>
>>
>> On Fri, Dec 11, 2015 at 12:45 PM, Roman Mohr <rm...@redhat.com> wrote:
>>
>>>
>>>
>>> On Fri, Dec 11, 2015 at 12:41 PM, Sandro Bonazzola <sbona...@redhat.com>
>>> wrote:
>>>
>>>> Hi,
>>>> oVirt 3.6.1 RC4 included latest bits allowing to auto import Hosted
>>>> Engine storage domain within the Engine.
>>>> In order to have a wider coverage of the Hosted Engine workflow we've
>>>> scheduled a Hosted Engine test day for next week
>>>> on Monday, December 14th.
>>>> Please join us testing Hosted Engine with oVirt 3.6.1 RC4, both on a
>>>> clean install and on an upgrade from 3.5 flow.
>>>>
>>>
>>> +1
>>>
>>> Will at least do a clean install with nfs on monday.
>>>
>>
>>
>> The hosted-engine-setup part worked for me. But activating the host which
>> was added by engine-setup failed. After the host was added to the cluster I
>> saw that it was in 'Non Operational' state.
>> Searching the logs I found
>>
>> > Message: Host hosted_engine_1 does not comply with the cluster Default
>> emulated machines. The current cluster compatibility level supports
>> [pc-i440fx-rhel7.2.0, pc-i440fx-2.1, pseries-rhel7.2.0] and the host
>> emulated machines are
>> pc-i440fx-rhel7.1.0,rhel6.3.0,pc-q35-rhel7.0.0,rhel6.1.0,rhel6.6.0,
>> rhel6.2.0,pc,pc-q35-rhel7.1.0,q35,rhel6.4.0,rhel6.0.0,rhel6.5.0,pc-i440fx-rhel7.0.0.
>>
>>
> your host emulated machine is "pc-i440fx-rhel7.1.0..." meaning either its
> RHEL 7.1 or qemu version isn't latest.
>
>
Ok thx, after updating qemu on the host I could activate it. Should not the
first host determine which emulated machines are supported in the cluster?

>
> My host is RHEL 7.1. After I tried to klick 'Reset emulated machines' on
>> the default cluster, the engine tried to activate the host but failed with
>> the following SQL exception on the AddExistingFileStorageDomainCommand:
>>
>> > 2015-12-14 16:25:40,336 ERROR
>> [org.ovirt.engine.core.bll.storage.AddExistingFileStorageDomainCommand]
>> (org.ovirt.thread.pool-8-thread-7) [789ca4b] Command
>> 'org.ovirt.engine.core.bll.sto
>> rage.AddExistingFileStorageDomainCommand' failed:
>> CallableStatementCallback; SQL [{call insertstorage_domain_static(?, ?, ?,
>> ?, ?, ?, ?, ?, ?, ?, ?, ?)}]; ERROR: null value in column "s torage"
>> violates not-null constraint
>> [...]
>>
>>
I guess we should not execute an invalid SQL statement when the host does
not comply with the cluster, right?


> Log is attached.
>>
>> I am using the following repo:
>> http://resources.ovirt.org/pub/yum-repo/mirrorlist-ovirt-3.6-pre-el7
>>
>> ovirt-engine version is 3.6.1.3-1.el7.
>>
>>
>>
>>> Thanks,
>>>> --
>>>> Sandro Bonazzola
>>>> Better technology. Faster innovation. Powered by community
>>>> collaboration.
>>>> See how it works at redhat.com
>>>>
>>>> ___
>>>> Devel mailing list
>>>> Devel@ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>
>>>
>>> Roman
>>>
>>
>> Best regards,
>> Roman
>>
>> ___
>> Users mailing list
>> us...@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>
Thanks,
Roman
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Proposal: Hystrix for realtime command monitoring

[Roman Mohr]

On Fri, Dec 11, 2015 at 10:28 AM, Roman Mohr <rm...@redhat.com> wrote:

> Hi,
>
> a status update, a request and a question:
>
> [...]
>
>>
>>>> A first implementation can be found on gerrit[3].
>>>>
>>>> The implementation should be almost ready. The new rpms for hystrix are
> also moving forward. I would need some people which can give me karma on
> bodhi. See be
>
>  [...]
>
>> # How to monitor the engine?
>>>>
>>>> It is as easy as starting a hystrix-dashboard [2] with
>>>>
>>>>   $ git clone https://github.com/Netflix/Hystrix.git
>>>>   $ cd Hystrix/hystrix-dashboard
>>>>   $ ../gradlew jettyRun
>>>>
>>>>
> As part of the hystrix rpms there is now also a 'hystrix-dashboard' rpm.
> Using it is pretty simple. Just install it with 'dnf install
> hystrix-dashboard' and start jetty with 'systemctrl start jetty'. Jetty
> will then listen on 8080 by default (and if you told selinux that jetty is
> allowed to access the network).
>
> [...]
>
>
>>>> # Security?
>>>>
>>>> In the provided patches the hystrix-metrics-servlet is accessible at
>>>> /ovirt-engine/api/hystrix.stream. It is protected by basic auth but
>>>> accessible
>>>> for everyone who can authenticate. We should probably restrict it to
>>>> admins.
>>>>
>>>> that would be great if it doesn't require too much work. If it does
>>>> then we can start with enabling/disabling via JMX using Roy's recent patch
>>>> [8]
>>>>
>>>>
> Since I had to implement JMX support anyway to enable and disable hystrix
> (disabled by default) I am wondering if I can remove the authentication
> part. There is no sensible data in the hystrix stream and all other
> services like the db health check are not protected either. It would make
> it again a little bit easier to use.
>
>  [...]
>
>> 3) Three unpackaged dependencies: archaius, hystrix-core, hystrix-contrib
>>>>
>>>>
>>>>
> All required packages will be available in rawhide the next few hours. All
> builds on koji succeeded.
> Also all packages for f23 were successfully build.
>
> I would appreciate if some of you find the time to give these f23 pacakges
> some karma:
>
> archaius-0.7.3-3.fc23
> <https://bodhi.fedoraproject.org/updates/FEDORA-2015-3ae4cc39c5> [9]
> (includes archaius-core and archaius-zookeeper)
> hystrix-1.4.21-4.fc23
> <https://bodhi.fedoraproject.org/updates/FEDORA-2015-35994552ed>  [10]
> (includes hystrix-core, hystrix-metrics-event-stream and hystrix-dashboard)
>
> On el7 I had to package a little bit more and the final hystrix package
> itself is still missing, but some karma on the first round of packages
> would be very helpful:
>
> archaius-0.4.1-1.el7
> <https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-dd72806724>
> [11] (includes archaius-core)
> mockito-1.9.0-19.el7
> <https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-7bf9b82936> [12]
> assertj-core-2.2.0-2.el7
> <https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-f02466a5da> [13]
> jctools-1.1-0.3.alpha.el7
> <https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-27b59f8bf2> [14]
> rxjava-1.0.13-2.el7
> <https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-37400bf69d>
> [15]
>
>

All additional packages for el7 are now also available on testing:

jackson-core-2.6.3-1.el7 [16]
<https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-b711a01041>
hystrix-1.4.21-5.el7
<https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-d770404d8b> [17]

As always, I am grateful for every karma!


>
>>>>
>>>>
>>>>
>>> Since you yesterday volunteered to package them I think this should not
>>>> stop us!:-)
>>>>
>>>> thanks a lot for the effort, I miss a proper analysis for s long.
>>>> Thanks for stepping up!
>>>>
>>>> michal
>>>>
>>>>
>>>> # References
>>>>
>>>> [1] https://github.com/Netflix/Hystrix
>>>> [2] https://github.com/Netflix/Hystrix/tree/master/hystrix-dashboard
>>>> [3] https://gerrit.ovirt.org/#/q/topic:hystrix
>>>> [4]
>>>> http://www.nurkiewicz.com/2015/02/storing-months-of-historical-metrics.html
>>>> [5]
>>>> https://github.com/Netflix/Hystrix/wiki/FAQ#what-is-the-processing-overhead-of-using-hystrix
>>>> [5] https://bugzilla.redhat.com

Re: [ovirt-devel] Proposal: Hystrix for realtime command monitoring

[Roman Mohr]

Hi,

a status update, a request and a question:

[...]

>
>>> A first implementation can be found on gerrit[3].
>>>
>>> The implementation should be almost ready. The new rpms for hystrix are
also moving forward. I would need some people which can give me karma on
bodhi. See be

 [...]

> # How to monitor the engine?
>>>
>>> It is as easy as starting a hystrix-dashboard [2] with
>>>
>>>   $ git clone https://github.com/Netflix/Hystrix.git
>>>   $ cd Hystrix/hystrix-dashboard
>>>   $ ../gradlew jettyRun
>>>
>>>
As part of the hystrix rpms there is now also a 'hystrix-dashboard' rpm.
Using it is pretty simple. Just install it with 'dnf install
hystrix-dashboard' and start jetty with 'systemctrl start jetty'. Jetty
will then listen on 8080 by default (and if you told selinux that jetty is
allowed to access the network).

[...]


>>> # Security?
>>>
>>> In the provided patches the hystrix-metrics-servlet is accessible at
>>> /ovirt-engine/api/hystrix.stream. It is protected by basic auth but
>>> accessible
>>> for everyone who can authenticate. We should probably restrict it to
>>> admins.
>>>
>>> that would be great if it doesn't require too much work. If it does then
>>> we can start with enabling/disabling via JMX using Roy's recent patch [8]
>>>
>>>
Since I had to implement JMX support anyway to enable and disable hystrix
(disabled by default) I am wondering if I can remove the authentication
part. There is no sensible data in the hystrix stream and all other
services like the db health check are not protected either. It would make
it again a little bit easier to use.

 [...]

> 3) Three unpackaged dependencies: archaius, hystrix-core, hystrix-contrib
>>>
>>>
>>>
All required packages will be available in rawhide the next few hours. All
builds on koji succeeded.
Also all packages for f23 were successfully build.

I would appreciate if some of you find the time to give these f23 pacakges
some karma:

archaius-0.7.3-3.fc23
 [9]
(includes archaius-core and archaius-zookeeper)
hystrix-1.4.21-4.fc23
  [10]
(includes hystrix-core, hystrix-metrics-event-stream and hystrix-dashboard)

On el7 I had to package a little bit more and the final hystrix package
itself is still missing, but some karma on the first round of packages
would be very helpful:

archaius-0.4.1-1.el7
 [11]
(includes archaius-core)
mockito-1.9.0-19.el7
 [12]
assertj-core-2.2.0-2.el7
 [13]
jctools-1.1-0.3.alpha.el7
 [14]
rxjava-1.0.13-2.el7
 [15]


>>>
>>>
>>>
>> Since you yesterday volunteered to package them I think this should not
>>> stop us!:-)
>>>
>>> thanks a lot for the effort, I miss a proper analysis for s long.
>>> Thanks for stepping up!
>>>
>>> michal
>>>
>>>
>>> # References
>>>
>>> [1] https://github.com/Netflix/Hystrix
>>> [2] https://github.com/Netflix/Hystrix/tree/master/hystrix-dashboard
>>> [3] https://gerrit.ovirt.org/#/q/topic:hystrix
>>> [4]
>>> http://www.nurkiewicz.com/2015/02/storing-months-of-historical-metrics.html
>>> [5]
>>> https://github.com/Netflix/Hystrix/wiki/FAQ#what-is-the-processing-overhead-of-using-hystrix
>>> [5] https://bugzilla.redhat.com/show_bug.cgi?id=1268216
>>> [6] https://bugzilla.redhat.com/show_bug.cgi?id=1268224
>>> [7] http://graphite.wikidot.com
>>>
>>> [8] https://gerrit.ovirt.org/#/c/29693/
>>>
>>>
[9]   https://bodhi.fedoraproject.org/updates/FEDORA-2015-3ae4cc39c5
[10] https://bodhi.fedoraproject.org/updates/FEDORA-2015-35994552ed
[11] https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-dd72806724
[12] https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-7bf9b82936
[13] https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-f02466a5da
[14] https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-27b59f8bf2
[15] https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-37400bf69d

___
>>> Devel mailing list
>>> Devel@ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>
>>>
>>>
>>
>> ___
>> Devel mailing list
>> Devel@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>
>
Thanks,
Roman
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] [ATN] Introduction of RESTAPI metamodel

[Roman Mohr]

On Tue, Oct 27, 2015 at 12:16 PM, Juan Hernández <jhern...@redhat.com>
wrote:

> On 10/27/2015 11:28 AM, Roman Mohr wrote:
> >
> >
> > On Tue, Oct 27, 2015 at 10:47 AM, Juan Hernández <jhern...@redhat.com
> > <mailto:jhern...@redhat.com>> wrote:
> >
> > On 10/27/2015 10:16 AM, Roman Mohr wrote:
> > >
> > >
> > > On Mon, Oct 26, 2015 at 5:32 PM, Juan Hernández <
> jhern...@redhat.com <mailto:jhern...@redhat.com>
> > > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>>> wrote:
> > >
> > > On 10/26/2015 04:56 PM, Roman Mohr wrote:
> > > > Hi Juan,
> > > >
> > > > The way to specify the contract look pretty clean and nice.
> > > > I would love to read a few words about the big picture. What
> > is the
> > > > final scenario?
> > > >
> > >
> > > The motivation for this change is that currently we don't have
> > a central
> > > place where the RESTAPI is specified, rather we have several
> > different
> > > places, using several different technologies:
> > >
> > > * XML schema for the data model.
> > > * JAX-RS for part of the operational model (without the
> > parameters).
> > > * rsdl_metadata.yaml for the parameters of the operational
> model.
> > >
> > > This makes it difficult to infer information about the model.
> For
> > > example, the generators of the SDKs have to download the XML
> > schema, and
> > > the RSDL (which is generated from the JAX-RS interfaces using
> > reflection
> > > and combining it with the information from the
> > rsdl_metadata.yaml file)
> > > and then they have to do their own computations to extract
> > what they
> > > need.
> > >
> > > Same happens with the CLI: it has to extract the information
> > it needs
> > > from the Python code generated for the Python SDK, yet another
> > level of
> > > indirection.
> > >
> > >
> > > You are right, that definitely needs to be cleaned up. I just want
> to
> > > discuss a few points below with you.
> > >
> > >
> > >
> > > We are also lacking a comprehensive reference documentation of
> the
> > > RESTAPI. What we currently have has been written by hand, and
> > gets out
> > > of sync very quickly, and we don't even notice.
> > >
> > >
> > > Did you also consider swagger? It is made for exactly that purpose.
> > > I created a demo in [1] which uses resteasy, weld,
> hibernate-validator
> > > and swagger to demonstrate how to do DRY with jaxrs.
> > > Would be great to hear you thoughts on that.
> > >
> > > And there is the great swagger-ui [8] to display the documentation
> > in a
> > > more human readable way.
> > >
> >
> > Yes, I considered Swagger, and rejected it because it is JSON
> centric,
> > and I think JSON isn't as good as Java to represent the contracts of
> our
> > RESTAPI.
> >
> >
> > You just write plain jax-rs, swagger just creates a description out of
> > it. So  the source defining the contract is pure java (jax-rs with some
> > swagger annotations for description, etc.).
> > Or am I missing the point here?
> >
>
> If I understand correctly the Swagger core is a JSON (or YAML)
> specification of the API. From that you can generate JAX-RS annotated
> code, not the other way around. So the specification document that you
> write is a JSON document.
>
> Alternatively, you can use the Swagger annotations to decorate your
> implementation, both the entity classes and the JAX-RS resource
> implementations, and then extract the model from that. But this is
> putting the implementation before the specification. That is where we
> are today, and it causes multiple problems. I think it is better to have
> the specification and the implementation separate. Swagger does that
> well when using JSON directly, our metamodel also does it well, but
> using a better language.
>
> >
> >
> > In addition we need to do these changes in a smooth way, without
> causing
> > big changes in th

Re: [ovirt-devel] [ATN] Introduction of RESTAPI metamodel

[Roman Mohr]

On Mon, Oct 26, 2015 at 5:32 PM, Juan Hernández <jhern...@redhat.com> wrote:

> On 10/26/2015 04:56 PM, Roman Mohr wrote:
> > Hi Juan,
> >
> > The way to specify the contract look pretty clean and nice.
> > I would love to read a few words about the big picture. What is the
> > final scenario?
> >
>
> The motivation for this change is that currently we don't have a central
> place where the RESTAPI is specified, rather we have several different
> places, using several different technologies:
>
> * XML schema for the data model.
> * JAX-RS for part of the operational model (without the parameters).
> * rsdl_metadata.yaml for the parameters of the operational model.
>
> This makes it difficult to infer information about the model. For
> example, the generators of the SDKs have to download the XML schema, and
> the RSDL (which is generated from the JAX-RS interfaces using reflection
> and combining it with the information from the rsdl_metadata.yaml file)
> and then they have to do their own computations to extract what they need.
>
> Same happens with the CLI: it has to extract the information it needs
> from the Python code generated for the Python SDK, yet another level of
> indirection.
>

You are right, that definitely needs to be cleaned up. I just want to
discuss a few points below with you.


>
> We are also lacking a comprehensive reference documentation of the
> RESTAPI. What we currently have has been written by hand, and gets out
> of sync very quickly, and we don't even notice.
>

Did you also consider swagger? It is made for exactly that purpose.
I created a demo in [1] which uses resteasy, weld, hibernate-validator and
swagger to demonstrate how to do DRY with jaxrs.
Would be great to hear you thoughts on that.

And there is the great swagger-ui [8] to display the documentation in a
more human readable way.


>
> To solve these issues I intend to have the specification of the RESTAPI
> only in one place, and using only one technology. I decided to use Java
> interfaces for that. Note however that they are just the support for the
> information, like paper is the support for ink. I decided to use Java
> because it is easy to create, modify and re-factor using tools familiar
> to most of us.
>
> These source of these interfaces is analysed (using QDox, currently) and
> a "model" of the RESTAPI is generated in memory. This model is
> independent of the supporting Java source, and easy to consume. For
> example, imagine that you want to list all the types available in the
> model and for each one display its documentation:
>
>   Model model = ...;
>   for (Type type : model.getTypes()) {
> Name name = type.getName();
> String doc = type.getDoc();
> System.out.println(name + ": " + doc);
>   }
>
> Something like this, but more elaborate, will be part of a web
> application that provides comprehensive reference documentation,
> assuming that we dedicate the time to write documentation comments in
> the specification.
>
> I intend to use this model also to do simplify the generators of the
> SDKs and the CLI.
>
> In addition these are some of the things that I would like to change in
> the near future (for 4.0):
>
> * Move the specification of the parameters of operations out of the
> rsdl_metadata.yaml file and into the model. For example:
>
>   @Service
>   public VmService {
> /**
>  * The operation to add a virtual machine.
>  */
> interface Add {
>   /**
>* The representation of the virtual machine is received
>* as parameter, and the representation of the created
>* virtual machine is returned as result.
>*/
>@In @Out Vm vm();
>
>/**
> * In the future, we will be able to specify other
> * parameters here.
> */
>@In Boolean force();
>
>/**
> * Even with default values.
> */
>@In default Boolean force() { return true; }
>
>/**
> * And we will be able to specify constraints, which
> * will replace the rsdl_metadata.yaml file.
> */
>@Constraint
>default boolean vmNameMustNotBeNull() {
>  return vm().name() != null;
>}
>  }
>   }
>
> * Enforce the constraints automatically. If the constraints are in the
> model, then we can just check them and reject requests before delivering
> them to the application. Currently we do this manually (and often
> forget) with calls to "validate(...)" methods.
>


Did you consider just annotating the DTOs with JSR-303 annotations and
integrate a validator with jax-rs?
See [2] for an example.


&

Re: [ovirt-devel] [ATN] Introduction of RESTAPI metamodel

[Roman Mohr]

On Tue, Oct 27, 2015 at 10:47 AM, Juan Hernández <jhern...@redhat.com>
wrote:

> On 10/27/2015 10:16 AM, Roman Mohr wrote:
> >
> >
> > On Mon, Oct 26, 2015 at 5:32 PM, Juan Hernández <jhern...@redhat.com
> > <mailto:jhern...@redhat.com>> wrote:
> >
> > On 10/26/2015 04:56 PM, Roman Mohr wrote:
> > > Hi Juan,
> > >
> > > The way to specify the contract look pretty clean and nice.
> > > I would love to read a few words about the big picture. What is the
> > > final scenario?
> > >
> >
> > The motivation for this change is that currently we don't have a
> central
> > place where the RESTAPI is specified, rather we have several
> different
> > places, using several different technologies:
> >
> > * XML schema for the data model.
> > * JAX-RS for part of the operational model (without the parameters).
> > * rsdl_metadata.yaml for the parameters of the operational model.
> >
> > This makes it difficult to infer information about the model. For
> > example, the generators of the SDKs have to download the XML schema,
> and
> > the RSDL (which is generated from the JAX-RS interfaces using
> reflection
> > and combining it with the information from the rsdl_metadata.yaml
> file)
> > and then they have to do their own computations to extract what they
> > need.
> >
> > Same happens with the CLI: it has to extract the information it needs
> > from the Python code generated for the Python SDK, yet another level
> of
> > indirection.
> >
> >
> > You are right, that definitely needs to be cleaned up. I just want to
> > discuss a few points below with you.
> >
> >
> >
> > We are also lacking a comprehensive reference documentation of the
> > RESTAPI. What we currently have has been written by hand, and gets
> out
> > of sync very quickly, and we don't even notice.
> >
> >
> > Did you also consider swagger? It is made for exactly that purpose.
> > I created a demo in [1] which uses resteasy, weld, hibernate-validator
> > and swagger to demonstrate how to do DRY with jaxrs.
> > Would be great to hear you thoughts on that.
> >
> > And there is the great swagger-ui [8] to display the documentation in a
> > more human readable way.
> >
>
> Yes, I considered Swagger, and rejected it because it is JSON centric,
> and I think JSON isn't as good as Java to represent the contracts of our
> RESTAPI.
>

You just write plain jax-rs, swagger just creates a description out of it.
So  the source defining the contract is pure java (jax-rs with some swagger
annotations for description, etc.).
Or am I missing the point here?


>
> In addition we need to do these changes in a smooth way, without causing
> big changes in the middle. For example, in the first step we need to
> preserve the JAX-RS interfaces as they are today, to avoid massive
> changes to all the resource implementations. This could be done with
>
Swagger, but would require custom code generators. With less effort we
> can do our own.
>

This is of course generally a difficult task. But I do not know why it
would be more difficult to write a custom swagger reader (if we even have
to, it can read the interfaces as well) .
They are pretty streight forward. Just look at [9], this contains the wole
jax-rs specific code to generate the swagger documentation.

But yes, I don't know every detail here of the engine and can't clearly say
that integrating that would just streight forward (my feeling tells me that
it would not be too hard). I am just under the impression that we would
benefit from that. Just reduces custom magic to a minimum.


>
> Swagger UI is certainly great. I did test it and it is really good. We
> may be able to copy some concepts.
>
> >
> >
> > To solve these issues I intend to have the specification of the
> RESTAPI
> > only in one place, and using only one technology. I decided to use
> Java
> > interfaces for that. Note however that they are just the support for
> the
> > information, like paper is the support for ink. I decided to use Java
> > because it is easy to create, modify and re-factor using tools
> familiar
> > to most of us.
> >
> > These source of these interfaces is analysed (using QDox, currently)
> and
> > a "model" of the RESTAPI is generated in memory. This model is
> > independent of the supporting Java source, and easy to consume. For
> > example, imagine that you want to list all the types 

Re: [ovirt-devel] [ATN] Introduction of RESTAPI metamodel

[Roman Mohr]

On Tue, Oct 27, 2015 at 1:45 PM, Juan Hernández <jhern...@redhat.com> wrote:

> On 10/27/2015 12:55 PM, Roman Mohr wrote:
> >
> >
> > On Tue, Oct 27, 2015 at 12:16 PM, Juan Hernández <jhern...@redhat.com
> > <mailto:jhern...@redhat.com>> wrote:
> >
> > On 10/27/2015 11:28 AM, Roman Mohr wrote:
> > >
> > >
> > > On Tue, Oct 27, 2015 at 10:47 AM, Juan Hernández <
> jhern...@redhat.com <mailto:jhern...@redhat.com>
> > > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>>> wrote:
> > >
> > > On 10/27/2015 10:16 AM, Roman Mohr wrote:
> > > >
> > > >
> > > > On Mon, Oct 26, 2015 at 5:32 PM, Juan Hernández <
> jhern...@redhat.com <mailto:jhern...@redhat.com>
> > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>>
> > > > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>
> > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>>>> wrote:
> > > >
> > > > On 10/26/2015 04:56 PM, Roman Mohr wrote:
> > > > > Hi Juan,
> > > > >
> > > > > The way to specify the contract look pretty clean and
> > nice.
> > > > > I would love to read a few words about the big
> > picture. What
> > > is the
> > > > > final scenario?
> > > > >
> > > >
> > > > The motivation for this change is that currently we
> > don't have
> > > a central
> > > > place where the RESTAPI is specified, rather we have
> several
> > > different
> > > > places, using several different technologies:
> > > >
> > > > * XML schema for the data model.
> > > > * JAX-RS for part of the operational model (without the
> > > parameters).
> > > > * rsdl_metadata.yaml for the parameters of the
> > operational model.
> > > >
> > > > This makes it difficult to infer information about the
> > model. For
> > > > example, the generators of the SDKs have to download the
> XML
> > > schema, and
> > > > the RSDL (which is generated from the JAX-RS interfaces
> > using
> > > reflection
> > > > and combining it with the information from the
> > > rsdl_metadata.yaml file)
> > > > and then they have to do their own computations to
> extract
> > > what they
> > > > need.
> > > >
> > > > Same happens with the CLI: it has to extract the
> information
> > > it needs
> > > > from the Python code generated for the Python SDK, yet
> > another
> > > level of
> > > > indirection.
> > > >
> > > >
> > > > You are right, that definitely needs to be cleaned up. I
> > just want to
> > > > discuss a few points below with you.
> > > >
> > > >
> > > >
> > > > We are also lacking a comprehensive reference
> > documentation of the
> > > > RESTAPI. What we currently have has been written by
> > hand, and
> > > gets out
> > > > of sync very quickly, and we don't even notice.
> > > >
> > > >
> > > > Did you also consider swagger? It is made for exactly that
> > purpose.
> > > > I created a demo in [1] which uses resteasy, weld,
> > hibernate-validator
> > > > and swagger to demonstrate how to do DRY with jaxrs.
> > > > Would be great to hear you thoughts on that.
> > > >
> > > > And there is the great swagger-ui [8] to display the
> > documentation
> > > in a
> > > > more human readable way.
> > > >
> > >
> > > Yes, I considered Swagger, and rejected it because it is JSON
> > centric,
> > > and I think JSON isn't as go

Re: [ovirt-devel] [ATN] Introduction of RESTAPI metamodel

[Roman Mohr]

On Tue, Oct 27, 2015 at 12:16 PM, Juan Hernández <jhern...@redhat.com>
wrote:

> On 10/27/2015 11:28 AM, Roman Mohr wrote:
> >
> >
> > On Tue, Oct 27, 2015 at 10:47 AM, Juan Hernández <jhern...@redhat.com
> > <mailto:jhern...@redhat.com>> wrote:
> >
> > On 10/27/2015 10:16 AM, Roman Mohr wrote:
> > >
> > >
> > > On Mon, Oct 26, 2015 at 5:32 PM, Juan Hernández <
> jhern...@redhat.com <mailto:jhern...@redhat.com>
> > > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>>> wrote:
> > >
> > > On 10/26/2015 04:56 PM, Roman Mohr wrote:
> > > > Hi Juan,
> > > >
> > > > The way to specify the contract look pretty clean and nice.
> > > > I would love to read a few words about the big picture. What
> > is the
> > > > final scenario?
> > > >
> > >
> > > The motivation for this change is that currently we don't have
> > a central
> > > place where the RESTAPI is specified, rather we have several
> > different
> > > places, using several different technologies:
> > >
> > > * XML schema for the data model.
> > > * JAX-RS for part of the operational model (without the
> > parameters).
> > > * rsdl_metadata.yaml for the parameters of the operational
> model.
> > >
> > > This makes it difficult to infer information about the model.
> For
> > > example, the generators of the SDKs have to download the XML
> > schema, and
> > > the RSDL (which is generated from the JAX-RS interfaces using
> > reflection
> > > and combining it with the information from the
> > rsdl_metadata.yaml file)
> > > and then they have to do their own computations to extract
> > what they
> > > need.
> > >
> > > Same happens with the CLI: it has to extract the information
> > it needs
> > > from the Python code generated for the Python SDK, yet another
> > level of
> > > indirection.
> > >
> > >
> > > You are right, that definitely needs to be cleaned up. I just want
> to
> > > discuss a few points below with you.
> > >
> > >
> > >
> > > We are also lacking a comprehensive reference documentation of
> the
> > > RESTAPI. What we currently have has been written by hand, and
> > gets out
> > > of sync very quickly, and we don't even notice.
> > >
> > >
> > > Did you also consider swagger? It is made for exactly that purpose.
> > > I created a demo in [1] which uses resteasy, weld,
> hibernate-validator
> > > and swagger to demonstrate how to do DRY with jaxrs.
> > > Would be great to hear you thoughts on that.
> > >
> > > And there is the great swagger-ui [8] to display the documentation
> > in a
> > > more human readable way.
> > >
> >
> > Yes, I considered Swagger, and rejected it because it is JSON
> centric,
> > and I think JSON isn't as good as Java to represent the contracts of
> our
> > RESTAPI.
> >
> >
> > You just write plain jax-rs, swagger just creates a description out of
> > it. So  the source defining the contract is pure java (jax-rs with some
> > swagger annotations for description, etc.).
> > Or am I missing the point here?
> >
>
> If I understand correctly the Swagger core is a JSON (or YAML)
> specification of the API. From that you can generate JAX-RS annotated
> code, not the other way around. So the specification document that you
> write is a JSON document.
>

You are right, my terminology here was not clear. Swagger is just a
specification. Swagger-core and swagger-jaxrs are the ones which can create
the documnetation out of JAX-RS resources.


> Alternatively, you can use the Swagger annotations to decorate your
> implementation, both the entity classes and the JAX-RS resource
> implementations, and then extract the model from that. But this is
> putting the implementation before the specification. That is where we
> are today, and it causes multiple problems. I think it is better to have
> the specification and the implementation separate. Swagger does that
> well when using JSON directly, our metamodel also does it wel

Re: [ovirt-devel] [ATN] Introduction of RESTAPI metamodel

[Roman Mohr]

On Tue, Oct 27, 2015 at 2:41 PM, Juan Hernández <jhern...@redhat.com> wrote:

> On 10/27/2015 02:10 PM, Roman Mohr wrote:
> >
> >
> > On Tue, Oct 27, 2015 at 1:45 PM, Juan Hernández <jhern...@redhat.com
> > <mailto:jhern...@redhat.com>> wrote:
> >
> > On 10/27/2015 12:55 PM, Roman Mohr wrote:
> > >
> > >
> > > On Tue, Oct 27, 2015 at 12:16 PM, Juan Hernández <
> jhern...@redhat.com <mailto:jhern...@redhat.com>
> > > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>>> wrote:
> > >
> > > On 10/27/2015 11:28 AM, Roman Mohr wrote:
> > > >
> > > >
> > > > On Tue, Oct 27, 2015 at 10:47 AM, Juan Hernández <
> jhern...@redhat.com <mailto:jhern...@redhat.com>
> > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>>
> > > > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>
> > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>>>> wrote:
> > > >
> > > > On 10/27/2015 10:16 AM, Roman Mohr wrote:
> > > > >
> > > > >
> > > > > On Mon, Oct 26, 2015 at 5:32 PM, Juan Hernández <
> jhern...@redhat.com <mailto:jhern...@redhat.com>
> > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>>
> > > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>
> > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>>>
> > > > > <mailto:jhern...@redhat.com  jhern...@redhat.com>
> > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>>
> > > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>
> > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>>>>> wrote:
> > > > >
> > > > > On 10/26/2015 04:56 PM, Roman Mohr wrote:
> > > > > > Hi Juan,
> > > > > >
> > > > > > The way to specify the contract look pretty
> > clean and
> > > nice.
> > > > > > I would love to read a few words about the big
> > > picture. What
> > > > is the
> > > > > > final scenario?
> > > > > >
> > > > >
> > > > > The motivation for this change is that currently we
> > > don't have
> > > > a central
> > > > > place where the RESTAPI is specified, rather we
> > have several
> > > > different
> > > > > places, using several different technologies:
> > > > >
> > > > > * XML schema for the data model.
> > > > > * JAX-RS for part of the operational model
> > (without the
> > > > parameters).
> > > > > * rsdl_metadata.yaml for the parameters of the
> > > operational model.
> > > > >
> > > > > This makes it difficult to infer information about
> the
> > > model. For
> > > > > example, the generators of the SDKs have to
> > download the XML
> > > > schema, and
> > > > > the RSDL (which is generated from the JAX-RS
> > interfaces
> > > using
> > > > reflection
> > > > > and combining it with the information from the
> > > > rsdl_metadata.yaml file)
> > > > > and then they have to do their own computations to
> > extract
> > > > what they
> > > > > need.
> > > > >
> > > > > Same happens with the CLI: it has to extract the
> > information
> > > > it needs
> > > > > from the Python code generated for the Python SDK,
> yet
> > > another
> > > > level of
> > > > > indirection.
> > > > >
> > >   

Re: [ovirt-devel] [ATN] Introduction of RESTAPI metamodel

[Roman Mohr]

On Tue, Oct 27, 2015 at 2:41 PM, Juan Hernández <jhern...@redhat.com> wrote:

> On 10/27/2015 02:10 PM, Roman Mohr wrote:
> >
> >
> > On Tue, Oct 27, 2015 at 1:45 PM, Juan Hernández <jhern...@redhat.com
> > <mailto:jhern...@redhat.com>> wrote:
> >
> > On 10/27/2015 12:55 PM, Roman Mohr wrote:
> > >
> > >
> > > On Tue, Oct 27, 2015 at 12:16 PM, Juan Hernández <
> jhern...@redhat.com <mailto:jhern...@redhat.com>
> > > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>>> wrote:
> > >
> > > On 10/27/2015 11:28 AM, Roman Mohr wrote:
> > > >
> > > >
> > > > On Tue, Oct 27, 2015 at 10:47 AM, Juan Hernández <
> jhern...@redhat.com <mailto:jhern...@redhat.com>
> > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>>
> > > > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>
> > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>>>> wrote:
> > > >
> > > > On 10/27/2015 10:16 AM, Roman Mohr wrote:
> > > > >
> > > > >
> > > > > On Mon, Oct 26, 2015 at 5:32 PM, Juan Hernández <
> jhern...@redhat.com <mailto:jhern...@redhat.com>
> > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>>
> > > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>
> > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>>>
> > > > > <mailto:jhern...@redhat.com  jhern...@redhat.com>
> > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>>
> > > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>
> > <mailto:jhern...@redhat.com <mailto:jhern...@redhat.com>>>>> wrote:
> > > > >
> > > > > On 10/26/2015 04:56 PM, Roman Mohr wrote:
> > > > > > Hi Juan,
> > > > > >
> > > > > > The way to specify the contract look pretty
> > clean and
> > > nice.
> > > > > > I would love to read a few words about the big
> > > picture. What
> > > > is the
> > > > > > final scenario?
> > > > > >
> > > > >
> > > > > The motivation for this change is that currently we
> > > don't have
> > > > a central
> > > > > place where the RESTAPI is specified, rather we
> > have several
> > > > different
> > > > > places, using several different technologies:
> > > > >
> > > > > * XML schema for the data model.
> > > > > * JAX-RS for part of the operational model
> > (without the
> > > > parameters).
> > > > > * rsdl_metadata.yaml for the parameters of the
> > > operational model.
> > > > >
> > > > > This makes it difficult to infer information about
> the
> > > model. For
> > > > > example, the generators of the SDKs have to
> > download the XML
> > > > schema, and
> > > > > the RSDL (which is generated from the JAX-RS
> > interfaces
> > > using
> > > > reflection
> > > > > and combining it with the information from the
> > > > rsdl_metadata.yaml file)
> > > > > and then they have to do their own computations to
> > extract
> > > > what they
> > > > > need.
> > > > >
> > > > > Same happens with the CLI: it has to extract the
> > information
> > > > it needs
> > > > > from the Python code generated for the Python SDK,
> yet
> > > another
> > > > level of
> > > > > indirection.
> > > > >
> > >   

Re: [ovirt-devel] [ATN] Introduction of RESTAPI metamodel

[Roman Mohr]

Hi Juan,

The way to specify the contract look pretty clean and nice.
I would love to read a few words about the big picture. What is the final
scenario?

On Mon, Oct 26, 2015 at 4:03 PM, Juan Hernández  wrote:

> Hello,
>
> I will soon merge the following patches that introduce a new way to
> specify the contracts of the RESTAPI:
>
>   restapi: Introduce metamodel
>   https://gerrit.ovirt.org/45852
>
>   restapi: Use metamodel
>   https://gerrit.ovirt.org/46478
>
>   restapi: Generate JAX-RS interfaces from model
>   https://gerrit.ovirt.org/47337
>
>
Looks pretty much like we are replacing one way of annotating things with
another way of specifying things.
Could you elaborate what the benefit of that way of description is?

How would I customize endpoints with e.g. @Gzip annotations? Would I at the
end still have my JAX-RS annotates resource classes?


> These patches introduce a new "metamodel" concept, and move the current
> specification of the RESTAPI based on XML schema and JAX-RS interfaces
> to a new "model" built on the new metamodel.
>

What does this mean for you in practical terms? Currently when you want
> to introduce or modify one of the data types used by the RESTAPI you
> start by modifying the XML schema. Once the patches are merged the XML
> schema will never be touched, as it will be automatically generated from
> the "model". For example, imagine that you need to add a new "color"
> attribute to the "VM" entity. To do so with the new model you will have
> to modify the following file, which is the specification of the "Vm"
> entity, written as a Java interface:
>
>
>
> https://gerrit.ovirt.org/#/c/46478/16/backend/manager/modules/restapi/model/src/main/java/types/Vm.java
>
> In that interface you will have to add a line like this:
>
>   String color();
>
> Note that this Java interface is just the specification of the entity,
> it won't be used at all during runtime. Instead of that the XML schema
> will be generated from it, and then Java will be generated from the XML
> schema, as we do today (this will change in the future, but not yet).
>
> Same for the services. If you want to add a new "paint" action to the
> "Vm" resource then you won't modify the JAX-RS interfaces, instead of
> that you will modify the following file, which is the specification of
> the "Vm" service, written as a Java interface:
>
>
>
> https://gerrit.ovirt.org/#/c/47337/6/backend/manager/modules/restapi/model/src/main/java/services/VmService.java
>
> In that interface you will need to add a sub-interface representing the
> action:
>
>   interface Paint {
>   }
>
> The JAX-RS interface will be generated from that. Currently these
> sub-interfaces are empty. In the future they will contain the
> specifications of the parameters (currently in the rsdl_metadata.yml file).


>
> These changes will currently affect only the specification of the
> RESTAPI, not the implementation, so in in the "Backend*Resource" classes
> things won't change yet.
>
>
Currently I do not really understand where we are going here. Are we trying
to get rid of rdsl?

So basically two questions:

1) What is the final goal?
2) What speaks agains using Hibernate validator on Daos in combination with
JAX-RS annotated resources (and just removing all interfaces, as far as I
can see we only have one implementation per endpoint) and creating all
schemas and clients through SWAGGER tooling?


> If you have doubts, please let me know.
>
> Regards,
> Juan Hernandez
>
> --
> Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta
> 3ºD, 28016 Madrid, Spain
> Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.
> ___
> Devel mailing list
> Devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel


Thanks,

Roman
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

[ovirt-devel] Proposal: Hystrix for realtime command monitoring

[Roman Mohr]

Hi All,

I am contributing to the engine for three months now. While I dug into the
code I
started to wonder how to visualize what the engine is actually doing.

To get better insights I added hystrix[1] to the engine. Hystrix is a
circuit
breaker library which was developed by Netflix and has one pretty
interesting
feature: Real time metrics for commands.

In combination with hystrix-dashboard[2] it allows very interesting
insights.
You can easily get an overview of commands involved in operations, their
performance and complexity. Look at [2] and the attachments in [5] and [6]
for
screenshots to get an Impression.

I want to propose to integrate hystrix permanently because from my
perspective
the results were really useful and I also had some good experiences with
hystrix
in past projects.

A first implementation can be found on gerrit[3].

# Where is it immediately useful?

During development and QA.

An example: I tested the hystrix integration on /api/vms and /api/hosts rest
endpoints and immediately saw that the number of command exectuions grew
lineary whit the number of vms and hosts. The bug reports [5] and [6] are
the
result.

# How to monitor the engine?

It is as easy as starting a hystrix-dashboard [2] with

  $ git clone https://github.com/Netflix/Hystrix.git
  $ cd Hystrix/hystrix-dashboard
  $ ../gradlew jettyRun

and point the dashboard to

   https:///ovirt-engine/hystrix.stream.

# Other possible benefits?

 * Live metrics at customer site for admins, consultants and support.

 * Historical metrics for analysis in addition to the log files.
   The metrics information is directly usable in graphite [7]. Therefore it
would be
   possible to collect the json stream for a certain time period and
analyze them
   later like in [4]. To do that someone just has to run

  curl --user admin@internal:engine
http://localhost:8080/ovirt-engine/api/hystrix.stream > hystrix.stream

   for as long as necessary. The results can be analyzed later.

# Possible architectural benefits?

In addition to the live metrics we might also have use for the real hystrix
features:

 * Circuit Breaker
 * Bulk execution of commands
 * De-dublication of commands (Caching)
 * Synchronous and asynchronous execution support
 * ...

Our commands do already have a lot of features, so I don't think that there
are
some quick wins, but maybe there are interesting opportunities for infra.

# Overhead?

In [5] the netflix employees describe their results regarding the overhead
of
wrapping every command into a new instance of a hystrix command.

They ran their tests on a standard 4-core Amazon EC2 server with a load of
60
request per second.

When using threadpools they measured a mean overhead of less than one
millisecond (so negligible).  At the 90th percentile they measured an
overhead
of 3 ms. At the 99th percentile of about 9 ms.

When configuring the hystrix commands to use semaphores instead of
threadpools
they are even faster.

# How to integrate?

A working implementation can be found on gerrit[3].  These patch sets wrap a
hystrix command around every VdcAction, every VdcQuery and every VDSCommand.
This just required four small modifications in the code base.

# Security?

In the provided patches the hystrix-metrics-servlet is accessible at
/ovirt-engine/api/hystrix.stream. It is protected by basic auth but
accessible
for everyone who can authenticate. We should probably restrict it to admins.

# Todo?

1) We do report failed actions with return values. Hystrix expects failing
commands to throw an exception.  So on the dashboard almost every command
looks
like a success.  To overcome this, it would be pretty easy to throw an
exception inside the command and catch it immediately after it leaves the
hystrix wrapper.

2) Finetuning
Do we want semaphores or a thread pool. When the thread pool, what size do
we want?

3) Three unpackaged dependencies: archaius, hystrix-core, hystrix-contrib

# References

[1] https://github.com/Netflix/Hystrix
[2] https://github.com/Netflix/Hystrix/tree/master/hystrix-dashboard
[3] https://gerrit.ovirt.org/#/q/topic:hystrix
[4]
http://www.nurkiewicz.com/2015/02/storing-months-of-historical-metrics.html
[5]
https://github.com/Netflix/Hystrix/wiki/FAQ#what-is-the-processing-overhead-of-using-hystrix
[5] https://bugzilla.redhat.com/show_bug.cgi?id=1268216
[6] https://bugzilla.redhat.com/show_bug.cgi?id=1268224
[7] http://graphite.wikidot.com
___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel

Re: [ovirt-devel] Vdsm: extending maintainers team

[Roman Mohr]

Gratulations! :-)
Am 09.08.2015 16:51 schrieb Allon Mureinik amure...@redhat.com:

 Congrats guys!


 On Sun, Aug 9, 2015 at 5:24 PM, Doron Fediuck dfedi...@redhat.com wrote:

 Congratulations all!

 On Thu, Aug 6, 2015 at 12:20 PM, Francesco Romani from...@redhat.com
 wrote:

 Thanks everyone!

 I'm honoured and very happy to have reached this milestone, and this is
 a very strong incentive to increase even further commitment!

 Bests,

 --

 *From: *Sandro Bonazzola sbona...@redhat.com
 *To: *Federico Simoncelli fsimo...@redhat.com, David Caro Estevez
 dc...@redhat.com, Eyal Edri ee...@redhat.com
 *Cc: *devel@ovirt.org
 *Sent: *Wednesday, August 5, 2015 1:48:34 PM
 *Subject: *Re: [ovirt-devel] Vdsm: extending maintainers team


 I think we had enough acks :-)

 On Wed, Aug 5, 2015 at 1:39 PM, Federico Simoncelli fsimo...@redhat.com
  wrote:

 On Tue, Aug 4, 2015 at 11:31 AM, Barak Azulay bazu...@redhat.com
 wrote:
 
  - Original Message -
  From: Dan Kenigsberg dan...@redhat.com
  To: devel@ovirt.org
  Cc: pklic...@redhat.com
  Sent: Tuesday, August 4, 2015 11:58:53 AM
  Subject: [ovirt-devel] Vdsm: extending maintainers team
 
  If you follow Vdsm development, you probably have noticed that we are
  short of active maintainers.
 
  Thankfully, we have have great developers that - in my opinion - can
  fill that gap. I am impressed by the quality of their reviews, their
  endurance, and most importantly - their ability to unbreak whatever
  code they approve.
 
  I'd like to nominate
  - Nir Soffer - for storage
 
  the above should be endorsed by Federico

 Endorsed! Good luck Nir!

  - Francesco Romani - for virt
 
  +1
 
  - Piotr Kliczewski - for infra
 
  +1

 Congrats guys!

 --
 Federico
 ___
 Devel mailing list
 Devel@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/devel




 --
 Sandro Bonazzola
 Better technology. Faster innovation. Powered by community collaboration.
 See how it works at redhat.com

 ___
 Devel mailing list
 Devel@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/devel




 --
 Francesco Romani
 RedHat Engineering Virtualization R  D
 Phone: 8261328
 IRC: fromani

 ___
 Devel mailing list
 Devel@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/devel



 ___
 Devel mailing list
 Devel@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/devel



 ___
 Devel mailing list
 Devel@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/devel

___
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel