Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

[ollivier . cedric]

Hello, 

Yes BUILD_TAG is published to the DB but of course no result will be
published by Functest if deployment is unsuccessful (UNTRIGGER)
https://build.opnfv.org/ci/view/functest/job/functest-daisy-baremetal-d
aily-fraser/lastSuccessfulBuild/injectedEnvVars/
http://xtesting.readthedocs.io/en/latest/apidoc/xtesting.core.testcase.
html#xtesting.core.testcase.TestCase.push_to_db

I would have thought that all data (Jenkins + DB) are already in place
to print results and even to highlight several CI/CD issues in OPNFV
(i.e. we can't verify test frameworks via the current model).
If you can't find a connection_check result via BUILD_TAG in DB, it
simply means that Functest (and all tests right after) has not been
called and the deployment is false (as returned to Jenkins)

Developping Jenkins macros (to publish the installer job results and
the red/blue balls from test frameworks) only increases the
maintainance:
Then it will work if Functest is not started right after the deployment
or if Functest changes testcase orders. 

As already highlighted, there is no diff in DB between ARM64 and AMD64
in case of Fuel.
You will have to pin the POD which is fully unsafe (ARM64 could switch
to other PODs).

Cédric

On jeu., 2018-05-03 at 03:32 +, Chenjiankun (JackChan) wrote:
> Hi David,
>  
> Thanks for your response. J
>  
> About the build_id, it is same as Jenkins macro, but I have checked
> the environment variable, I don’t see any variable can be used. Maybe
> I have missing something? @Julien, can you give some clarification?
>  
> For the health check and fuel@x86@aarm64 issue, I think it is
> not had to implement. I will add them in another patches. We can
> improve this page step by step.
>  
> BRs,
> Jack Chan
>  
> 发件人: David McBride [mailto:dmcbr...@linuxfoundation.org] 
> 发送时间: 2018年4月26日 11:53
> 收件人: Chenjiankun (JackChan)
> 抄送: TECH-DISCUSS OPNFV; tro...@redhat.com; Brattain, Ross B; Rao,
> Sridhar; OLLIVIER Cédric IMT/OLN; mark.bei...@dell.com; Yuyang
> (Gabriel); ALFRED C 'MORTON ' (acmor...@att.com);
> emma.l.fo...@intel.com; Liyin (Ace); Wangwulin (Linda); georg.kunz@er
> icsson.com; Serena Feng; Julien
> 主题: Re: [opnfv-tech-discuss][test-wg]Requirements for test resources
> collection
>  
> Hi Jack,
>  
> Sorry for the slow response.  I've been busy with the Fraser release.
>  
> Comments
> I didn't understand point 1 about the build_id.  I thought that was
> the purpose of the Jenkins macro that Julien developed.  What am I
> missing?
> I'd like to see Health Check broken out as a separate column from
> Functest.
> The installer drop-down just lists "Fuel".  We need to include both
> Fuel@x86 and Fuel@aarm64.
> Thanks for your effort in pushing this forward.
>  
> David
>  
>  
> On Sun, Apr 22, 2018 at 8:49 PM, Chenjiankun (JackChan)
> <chenjiank...@huawei.com> wrote:
> Hi David,
>  
> As discussed in the previous Email, I have create a first version of 
> scenario results table(we can call it gating reporting J), see:
>  http://116.66.187.136:9998/index.html
> Patch see:
>  https://gerrit.opnfv.org/gerrit/#/c/56179/
>  
> For now, I have added four filters:
> 1.   Scenario
> 
> 2.   Version
> 
> 3.   Installer
> 
> 4.   Iteration(we will query the last 10 days data, the default
> iteration is 10, if all record is more than 10, it will only 10
> record)
> 
>  
> For now, we still have some problems to solve:
> 1. build_id(Jenkins id), we don’t have a unified ID to mark the
> whole Jenkins jobs(deployment + functest + yardstick), for now, I use
> the build_tag(e.g. jenkins-functest-fuel-baremetal-daily-master-131,
> the project:functest and installer:fuel are in build_tag) to index,
> so we can only show the functest result.
> 
> 2.   Jenkins job results, depend on build_id, we can’t show all
> results(deployment + functest + yardstick, only functest for now).
> And each projects need to upload their results(UNTRIGGERED, PASS,
> FAIL).
> 
> 3.   Deployment results, for now, only daisy upload its
> deployment result, we need ask each installer to upload deployment
> results, only build_id ready, we can show all results.
> 
> 4.   Statistic results, depend on projects upload their results
> 
>  
> It is the first version, we will add more functions step by step
> after the above issue solved.
> @All, if you have any suggestions, please let me know.
>  
> BRs,
> Jack Chan
>  
> 发件人: chenjiankun 
> 发送时间: 2018年1月22日 17:02
> 收件人: 'David McBride'
> 抄送: TECH-DISCUSS OPNFV; tro...@redhat.com; Brattain, Ross B; Rao,
> Sridhar; OLLIVIER Cédric IMT/OLN; mark.bei...@dell.com; Yuyang
> (Gabriel); ALFRED C 'MORTON ' (acmor...@att.com);
> emma.l.fo...@int

Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

[Chenjiankun (JackChan)]

Hi David,

Thanks for your response. ☺

About the build_id, it is same as Jenkins macro, but I have checked the 
environment variable, I don’t see any variable can be used. Maybe I have 
missing something? @Julien, can you give some clarification?

For the health check and fuel@x86@aarm64 issue, I think it is not had to 
implement. I will add them in another patches. We can improve this page step by 
step.

BRs,
Jack Chan

发件人: David McBride [mailto:dmcbr...@linuxfoundation.org]
发送时间: 2018年4月26日 11:53
收件人: Chenjiankun (JackChan)
抄送: TECH-DISCUSS OPNFV; tro...@redhat.com; Brattain, Ross B; Rao, Sridhar; 
OLLIVIER Cédric IMT/OLN; mark.bei...@dell.com; Yuyang (Gabriel); ALFRED C 
'MORTON ' (acmor...@att.com); emma.l.fo...@intel.com; Liyin (Ace); Wangwulin 
(Linda); georg.k...@ericsson.com; Serena Feng; Julien
主题: Re: [opnfv-tech-discuss][test-wg]Requirements for test resources collection

Hi Jack,

Sorry for the slow response.  I've been busy with the Fraser release.

Comments

  1.  I didn't understand point 1 about the build_id.  I thought that was the 
purpose of the Jenkins macro that Julien developed.  What am I missing?
  2.  I'd like to see Health Check broken out as a separate column from 
Functest.
  3.  The installer drop-down just lists "Fuel".  We need to include both 
Fuel@x86 and Fuel@aarm64.
Thanks for your effort in pushing this forward.

David


On Sun, Apr 22, 2018 at 8:49 PM, Chenjiankun (JackChan) 
<chenjiank...@huawei.com<mailto:chenjiank...@huawei.com>> wrote:
Hi David,

As discussed in the previous Email, I have create a first version of  scenario 
results table(we can call it gating reporting ☺), see:
 http://116.66.187.136:9998/index.html
Patch see:
 https://gerrit.opnfv.org/gerrit/#/c/56179/

For now, I have added four filters:

1.   Scenario

2.   Version

3.   Installer

4.   Iteration(we will query the last 10 days data, the default iteration 
is 10, if all record is more than 10, it will only 10 record)

For now, we still have some problems to solve:

1. build_id(Jenkins id), we don’t have a unified ID to mark the whole 
Jenkins jobs(deployment + functest + yardstick), for now, I use the 
build_tag(e.g. jenkins-functest-fuel-baremetal-daily-master-131, the 
project:functest and installer:fuel are in build_tag) to index, so we can only 
show the functest result.

2.   Jenkins job results, depend on build_id, we can’t show all 
results(deployment + functest + yardstick, only functest for now). And each 
projects need to upload their results(UNTRIGGERED, PASS, FAIL).

3.   Deployment results, for now, only daisy upload its deployment result, 
we need ask each installer to upload deployment results, only build_id ready, 
we can show all results.

4.   Statistic results, depend on projects upload their results

It is the first version, we will add more functions step by step after the 
above issue solved.
@All, if you have any suggestions, please let me know.

BRs,
Jack Chan

发件人: chenjiankun
发送时间: 2018年1月22日 17:02
收件人: 'David McBride'
抄送: TECH-DISCUSS OPNFV; tro...@redhat.com<mailto:tro...@redhat.com>; Brattain, 
Ross B; Rao, Sridhar; OLLIVIER Cédric IMT/OLN; 
mark.bei...@dell.com<mailto:mark.bei...@dell.com>; Yuyang (Gabriel); ALFRED C 
'MORTON ' (acmor...@att.com<mailto:acmor...@att.com>); 
emma.l.fo...@intel.com<mailto:emma.l.fo...@intel.com>; Liyin (Ace); Wangwulin 
(Linda); georg.k...@ericsson.com<mailto:georg.k...@ericsson.com>; Serena Feng; 
Julien
主题: RE: [opnfv-tech-discuss][test-wg]Requirements for test resources collection

Thanks, David.

According your descriptions, I have created a demo table as below(wish I do not 
misunderstanding your meaning):

scenario

date

Jenkins

Version

Installer

Deployment

Functest

yardstick

os-nosdn-nofeature-ha

2018-01-21

Jenkins id

euphrates

compass

pass

pass

pass


2018-01-21

Jenkins id

euphrates

compass

fail

not trigger

not trigger


statistic

8/9/10
(pass:8,triggered:9, total:10)

6/7/8

6/7/8



This last line in table body is the statistics information, and lines above are 
the detailed information(and it can be folded).
The score 8/9/10 is pass/triggered/total. Total means should run, triggered 
means actually run.
Also we can add three filters:

If you select installer as compass, then will show all data related to compass.
Iterations means last x data points to be displayed.

Does this table satisfied your requirements?

BRs,
Jack Chan
发件人: David McBride [mailto:dmcbr...@linuxfoundation.org]
发送时间: 2018年1月20日 3:07
收件人: chenjiankun
抄送: TECH-DISCUSS OPNFV; tro...@redhat.com<mailto:tro...@redhat.com>; Brattain, 
Ross B; Rao, Sridhar; OLLIVIER Cédric IMT/OLN; 
mark.bei...@dell.com<mailto:mark.bei...@dell.com>; Yuyang (Gabriel); ALFRED C 
'MORTON ' (acmor...@att.com<mailto:acmor...@att.com>); 
emma.l.fo...@intel.com<mailto:emma.l.fo...@intel.com>; Liyin (Ace); Wangwulin 
(Linda); georg.k.

Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

[Julien]

Hi,

For we don't have enough resources for Jenkins logs or other reasons, for
example, they are not valuable any more, the Jenkins system may delete some
of the historical records. Someday, you may find that the record indicated
by build_id won't exist in the future. It is OK if it does not exist in my
opinion.

Julien,
BR

David McBride <dmcbr...@linuxfoundation.org>于2018年4月26日周四 上午11:53写道：

> Hi Jack,
>
> Sorry for the slow response.  I've been busy with the Fraser release.
>
> Comments
>
>1. I didn't understand point 1 about the build_id.  I thought that was
>the purpose of the Jenkins macro that Julien developed.  What am I missing?
>2. I'd like to see Health Check broken out as a separate column from
>Functest.
>3. The installer drop-down just lists "Fuel".  We need to include both
>Fuel@x86 and Fuel@aarm64.
>
> Thanks for your effort in pushing this forward.
>
> David
>
>
> On Sun, Apr 22, 2018 at 8:49 PM, Chenjiankun (JackChan) <
> chenjiank...@huawei.com> wrote:
>
>> Hi David,
>>
>>
>>
>> As discussed in the previous Email, I have create a first version of
>> scenario results table(we can call it gating reporting J), see:
>>
>>  http://116.66.187.136:9998/index.html
>>
>> Patch see:
>>
>>  https://gerrit.opnfv.org/gerrit/#/c/56179/
>>
>>
>>
>> For now, I have added four filters:
>>
>> 1.   Scenario
>>
>> 2.   Version
>>
>> 3.   Installer
>>
>> 4.   Iteration(we will query the last 10 days data, the default
>> iteration is 10, if all record is more than 10, it will only 10 record)
>>
>>
>>
>> For now, we still have some problems to solve:
>>
>> 1. build_id(Jenkins id), we don’t have a unified ID to mark the
>> whole Jenkins jobs(deployment + functest + yardstick), for now, I use the
>> build_tag(e.g. jenkins-functest-fuel-baremetal-daily-master-131, the
>> project:functest and installer:fuel are in build_tag) to index, so we can
>> only show the functest result.
>>
>> 2.   Jenkins job results, depend on build_id, we can’t show all
>> results(deployment + functest + yardstick, only functest for now). And each
>> projects need to upload their results(UNTRIGGERED, PASS, FAIL).
>>
>> 3.   Deployment results, for now, only daisy upload its deployment
>> result, we need ask each installer to upload deployment results, only
>> build_id ready, we can show all results.
>>
>> 4.   Statistic results, depend on projects upload their results
>>
>>
>>
>> It is the first version, we will add more functions step by step after
>> the above issue solved.
>>
>> @All, if you have any suggestions, please let me know.
>>
>>
>>
>> BRs,
>>
>> Jack Chan
>>
>>
>>
>> *发件人:* chenjiankun
>> *发送时间:* 2018年1月22日 17:02
>> *收件人:* 'David McBride'
>> *抄送:* TECH-DISCUSS OPNFV; tro...@redhat.com; Brattain, Ross B; Rao,
>> Sridhar; OLLIVIER Cédric IMT/OLN; mark.bei...@dell.com; Yuyang
>> (Gabriel); ALFRED C 'MORTON ' (acmor...@att.com); emma.l.fo...@intel.com;
>> Liyin (Ace); Wangwulin (Linda); georg.k...@ericsson.com; Serena Feng;
>> Julien
>> *主题:* RE: [opnfv-tech-discuss][test-wg]Requirements for test resources
>> collection
>>
>>
>>
>> Thanks, David.
>>
>>
>>
>> According your descriptions, I have created a demo table as below(wish I
>> do not misunderstanding your meaning):
>>
>>
>>
>> *scenario *
>>
>> *date *
>>
>> *Jenkins *
>>
>> *Version *
>>
>> *Installer *
>>
>> *Deployment *
>>
>> *Functest *
>>
>> *yardstick *
>>
>> os-nosdn-nofeature-ha
>>
>> 2018-01-21
>>
>> Jenkins id
>>
>> euphrates
>>
>> compass
>>
>> pass
>>
>> pass
>>
>> pass
>>
>> 2018-01-21
>>
>> Jenkins id
>>
>> euphrates
>>
>> compass
>>
>> fail
>>
>> not trigger
>>
>> not trigger
>>
>> statistic
>>
>> 8/9/10
>>
>> (pass:8,triggered:9, total:10)
>>
>> 6/7/8
>>
>> 6/7/8
>>
>>
>>
>>
>>
>> This last line in table body is the statistics information, and lines
>> above are the detailed information(and it can be folded).
>>
>> The score 8/9/10 is pass/triggered/total. Total means should run,
>> triggered means actually run.
>>
>>

Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

[David McBride]

Hi Jack,

Sorry for the slow response.  I've been busy with the Fraser release.

Comments

   1. I didn't understand point 1 about the build_id.  I thought that was
   the purpose of the Jenkins macro that Julien developed.  What am I missing?
   2. I'd like to see Health Check broken out as a separate column from
   Functest.
   3. The installer drop-down just lists "Fuel".  We need to include both
   Fuel@x86 and Fuel@aarm64.

Thanks for your effort in pushing this forward.

David


On Sun, Apr 22, 2018 at 8:49 PM, Chenjiankun (JackChan) <
chenjiank...@huawei.com> wrote:

> Hi David,
>
>
>
> As discussed in the previous Email, I have create a first version of
> scenario results table(we can call it gating reporting J), see:
>
>  http://116.66.187.136:9998/index.html
>
> Patch see:
>
>  https://gerrit.opnfv.org/gerrit/#/c/56179/
>
>
>
> For now, I have added four filters:
>
> 1.   Scenario
>
> 2.   Version
>
> 3.   Installer
>
> 4.   Iteration(we will query the last 10 days data, the default
> iteration is 10, if all record is more than 10, it will only 10 record)
>
>
>
> For now, we still have some problems to solve:
>
> 1. build_id(Jenkins id), we don’t have a unified ID to mark the whole
> Jenkins jobs(deployment + functest + yardstick), for now, I use the
> build_tag(e.g. jenkins-functest-fuel-baremetal-daily-master-131, the
> project:functest and installer:fuel are in build_tag) to index, so we can
> only show the functest result.
>
> 2.   Jenkins job results, depend on build_id, we can’t show all
> results(deployment + functest + yardstick, only functest for now). And each
> projects need to upload their results(UNTRIGGERED, PASS, FAIL).
>
> 3.   Deployment results, for now, only daisy upload its deployment
> result, we need ask each installer to upload deployment results, only
> build_id ready, we can show all results.
>
> 4.   Statistic results, depend on projects upload their results
>
>
>
> It is the first version, we will add more functions step by step after the
> above issue solved.
>
> @All, if you have any suggestions, please let me know.
>
>
>
> BRs,
>
> Jack Chan
>
>
>
> *发件人:* chenjiankun
> *发送时间:* 2018年1月22日 17:02
> *收件人:* 'David McBride'
> *抄送:* TECH-DISCUSS OPNFV; tro...@redhat.com; Brattain, Ross B; Rao,
> Sridhar; OLLIVIER Cédric IMT/OLN; mark.bei...@dell.com; Yuyang (Gabriel);
> ALFRED C 'MORTON ' (acmor...@att.com); emma.l.fo...@intel.com; Liyin
> (Ace); Wangwulin (Linda); georg.k...@ericsson.com; Serena Feng; Julien
> *主题:* RE: [opnfv-tech-discuss][test-wg]Requirements for test resources
> collection
>
>
>
> Thanks, David.
>
>
>
> According your descriptions, I have created a demo table as below(wish I
> do not misunderstanding your meaning):
>
>
>
> *scenario *
>
> *date *
>
> *Jenkins *
>
> *Version *
>
> *Installer *
>
> *Deployment *
>
> *Functest *
>
> *yardstick *
>
> os-nosdn-nofeature-ha
>
> 2018-01-21
>
> Jenkins id
>
> euphrates
>
> compass
>
> pass
>
> pass
>
> pass
>
> 2018-01-21
>
> Jenkins id
>
> euphrates
>
> compass
>
> fail
>
> not trigger
>
> not trigger
>
> statistic
>
> 8/9/10
>
> (pass:8,triggered:9, total:10)
>
> 6/7/8
>
> 6/7/8
>
>
>
>
>
> This last line in table body is the statistics information, and lines
> above are the detailed information(and it can be folded).
>
> The score 8/9/10 is pass/triggered/total. Total means should run,
> triggered means actually run.
>
> Also we can add three filters:
>
>
>
> If you select installer as compass, then will show all data related to
> compass.
>
> Iterations means last x data points to be displayed.
>
>
>
> Does this table satisfied your requirements?
>
>
>
> BRs,
>
> Jack Chan
>
> *发件人:* David McBride [mailto:dmcbr...@linuxfoundation.org
> <dmcbr...@linuxfoundation.org>]
> *发送时间:* 2018年1月20日 3:07
> *收件人:* chenjiankun
> *抄送:* TECH-DISCUSS OPNFV; tro...@redhat.com; Brattain, Ross B; Rao,
> Sridhar; OLLIVIER Cédric IMT/OLN; mark.bei...@dell.com; Yuyang (Gabriel);
> ALFRED C 'MORTON ' (acmor...@att.com); emma.l.fo...@intel.com; Liyin
> (Ace); Wangwulin (Linda); georg.k...@ericsson.com; Serena Feng; Julien
> *主题:* Re: [opnfv-tech-discuss][test-wg]Requirements for test resources
> collection
>
>
>
> +Serena, Julien
>
>
>
> Thanks, Jack.
>
>1. Data reported per scenario (i.e., jenkins job, deployment,
>functest, yardstick, etc. displayed together for each scenario) instead of
>

Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

[Chenjiankun (JackChan)]

Hi David,

As discussed in the previous Email, I have create a first version of  scenario 
results table(we can call it gating reporting ☺), see:
 http://116.66.187.136:9998/index.html
Patch see:
 https://gerrit.opnfv.org/gerrit/#/c/56179/

For now, I have added four filters:

1.   Scenario

2.   Version

3.   Installer

4.   Iteration(we will query the last 10 days data, the default iteration 
is 10, if all record is more than 10, it will only 10 record)

For now, we still have some problems to solve:

1. build_id(Jenkins id), we don’t have a unified ID to mark the whole 
Jenkins jobs(deployment + functest + yardstick), for now, I use the 
build_tag(e.g. jenkins-functest-fuel-baremetal-daily-master-131, the 
project:functest and installer:fuel are in build_tag) to index, so we can only 
show the functest result.

2.   Jenkins job results, depend on build_id, we can’t show all 
results(deployment + functest + yardstick, only functest for now). And each 
projects need to upload their results(UNTRIGGERED, PASS, FAIL).

3.   Deployment results, for now, only daisy upload its deployment result, 
we need ask each installer to upload deployment results, only build_id ready, 
we can show all results.

4.   Statistic results, depend on projects upload their results

It is the first version, we will add more functions step by step after the 
above issue solved.
@All, if you have any suggestions, please let me know.

BRs,
Jack Chan

发件人: chenjiankun
发送时间: 2018年1月22日 17:02
收件人: 'David McBride'
抄送: TECH-DISCUSS OPNFV; tro...@redhat.com; Brattain, Ross B; Rao, Sridhar; 
OLLIVIER Cédric IMT/OLN; mark.bei...@dell.com; Yuyang (Gabriel); ALFRED C 
'MORTON ' (acmor...@att.com); emma.l.fo...@intel.com; Liyin (Ace); Wangwulin 
(Linda); georg.k...@ericsson.com; Serena Feng; Julien
主题: RE: [opnfv-tech-discuss][test-wg]Requirements for test resources collection

Thanks, David.

According your descriptions, I have created a demo table as below(wish I do not 
misunderstanding your meaning):

scenario

date

Jenkins

Version

Installer

Deployment

Functest

yardstick

os-nosdn-nofeature-ha

2018-01-21

Jenkins id

euphrates

compass

pass

pass

pass


2018-01-21

Jenkins id

euphrates

compass

fail

not trigger

not trigger


statistic

8/9/10
(pass:8,triggered:9, total:10)

6/7/8

6/7/8



This last line in table body is the statistics information, and lines above are 
the detailed information(and it can be folded).
The score 8/9/10 is pass/triggered/total. Total means should run, triggered 
means actually run.
Also we can add three filters:

If you select installer as compass, then will show all data related to compass.
Iterations means last x data points to be displayed.

Does this table satisfied your requirements?

BRs,
Jack Chan
发件人: David McBride [mailto:dmcbr...@linuxfoundation.org]
发送时间: 2018年1月20日 3:07
收件人: chenjiankun
抄送: TECH-DISCUSS OPNFV; tro...@redhat.com<mailto:tro...@redhat.com>; Brattain, 
Ross B; Rao, Sridhar; OLLIVIER Cédric IMT/OLN; 
mark.bei...@dell.com<mailto:mark.bei...@dell.com>; Yuyang (Gabriel); ALFRED C 
'MORTON ' (acmor...@att.com<mailto:acmor...@att.com>); 
emma.l.fo...@intel.com<mailto:emma.l.fo...@intel.com>; Liyin (Ace); Wangwulin 
(Linda); georg.k...@ericsson.com<mailto:georg.k...@ericsson.com>; Serena Feng; 
Julien
主题: Re: [opnfv-tech-discuss][test-wg]Requirements for test resources collection

+Serena, Julien

Thanks, Jack.

  1.  Data reported per scenario (i.e., jenkins job, deployment, functest, 
yardstick, etc. displayed together for each scenario) instead of separate test 
silos.
  2.  Include deployment results
  3.  Include all Jenkins job results (failure to start, failure to complete, 
etc.)
  4.  Clear date/time stamps for every data point
  5.  Display the data above for the last x data points (e.g., 4, 5, 10 ?)
  6.  Use an easy-to-understand, unified scoring method for all test frameworks.
As I mentioned, yesterday, Julien and Serena have been working on this, as 
well.  Julien has developed a macro<https://gerrit.opnfv.org/gerrit/#/c/48515/> 
to enable consolidation of all results per scenario. He is intending to use the 
Daisy installer as a platform to verify the macro, which then can be adapted to 
other installers.

In addition, Serena has agreed to help manage an intern who can assist with the 
project.  I have an action to create an intern proposal for that purpose.

David

On Fri, Jan 19, 2018 at 1:23 AM, chenjiankun 
<chenjiank...@huawei.com<mailto:chenjiank...@huawei.com>> wrote:
Hi,

As we discussed last test working group weekly meeting, we want to do test 
resources aggregation.
We plan to offer a new friendly web portal which contain all existing test 
resource and more functions.

I have a broad classification as bellow:

1.   Data analysis

a) Reporting(existing, For release)

b) Bitergia(existing)

c) Grafana(existing, For detailed test r

Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

[David McBride]

+Fatih

On Mon, Jan 22, 2018 at 2:08 AM, SerenaFeng(zte) <
serena.feng.711+...@gmail.com> wrote:

> Hi David & Jack,
>
> Thanks for mentioning the task, an interface for collecting deployment
> results has been ready in TestAPI[1],
> the Macro for pushing deployment result is also ready by Jelien[2], I
> believe it will facilitate all the installers' work,
> currently, Julien is working on pushing daisy results to TestAPI
> leveraging that Macro, I think it will be finished soon.
>
> And regarding how to show all the information in a table, I suggest we can
> take a look at Jack's proposal first.
>
> @Jack, a few comments:
> 1. what's the opinion of jenkins id?
> 2. for a scenario-installer combination, some will not run once in a
> day(trigger multiple times or run in multiple pods),
> in this case, a simple pass/fail will be too vague, and to facilitate
> the support of data iteration, I would suggest
> leveraging 8/9/10(8 passed, 9 triggered, 10total), delete the
> final statistic line
> 3. how about adding a healthcheck column(functest-healthcheck test cases),
> to see if the installer meets the milestone 3.0
>
>
> [1]: https://gerrit.opnfv.org/gerrit/#/c/49895/
> [2]: https://gerrit.opnfv.org/gerrit/#/c/48515/
>
> BRs
> Serena
>
> On Mon, Jan 22, 2018 at 5:02 PM chenjiankun <chenjiank...@huawei.com>
> wrote:
>
>> Thanks, David.
>>
>>
>>
>> According your descriptions, I have created a demo table as below(wish I
>> do not misunderstanding your meaning):
>>
>>
>>
>> *scenario *
>>
>> *date *
>>
>> *Jenkins *
>>
>> *Version *
>>
>> *Installer *
>>
>> *Deployment *
>>
>> *Functest *
>>
>> *yardstick *
>>
>> os-nosdn-nofeature-ha
>>
>> 2018-01-21
>>
>> Jenkins id
>>
>> euphrates
>>
>> compass
>>
>> pass
>>
>> pass
>>
>> pass
>>
>> 2018-01-21
>>
>> Jenkins id
>>
>> euphrates
>>
>> compass
>>
>> fail
>>
>> not trigger
>>
>> not trigger
>>
>> statistic
>>
>> 8/9/10
>>
>> (pass:8,triggered:9, total:10)
>>
>> 6/7/8
>>
>> 6/7/8
>>
>>
>>
>>
>>
>> This last line in table body is the statistics information, and lines
>> above are the detailed information(and it can be folded).
>>
>> The score 8/9/10 is pass/triggered/total. Total means should run,
>> triggered means actually run.
>>
>> Also we can add three filters:
>>
>>
>>
>> If you select installer as compass, then will show all data related to
>> compass.
>>
>> Iterations means last x data points to be displayed.
>>
>>
>>
>> Does this table satisfied your requirements?
>>
>>
>>
>> BRs,
>>
>> Jack Chan
>>
>> *发件人:* David McBride [mailto:dmcbr...@linuxfoundation.org]
>> *发送时间:* 2018年1月20日 3:07
>> *收件人:* chenjiankun
>> *抄送:* TECH-DISCUSS OPNFV; tro...@redhat.com; Brattain, Ross B; Rao,
>> Sridhar; OLLIVIER Cédric IMT/OLN; mark.bei...@dell.com; Yuyang
>> (Gabriel); ALFRED C 'MORTON ' (acmor...@att.com); emma.l.fo...@intel.com;
>> Liyin (Ace); Wangwulin (Linda); georg.k...@ericsson.com; Serena Feng;
>> Julien
>> *主题:* Re: [opnfv-tech-discuss][test-wg]Requirements for test resources
>> collection
>>
>>
>>
>> +Serena, Julien
>>
>>
>>
>> Thanks, Jack.
>>
>>1. Data reported per scenario (i.e., jenkins job, deployment,
>>functest, yardstick, etc. displayed together for each scenario) instead of
>>separate test silos.
>>2. Include deployment results
>>3. Include all Jenkins job results (failure to start, failure to
>>complete, etc.)
>>4. Clear date/time stamps for every data point
>>5. Display the data above for the last x data points (e.g., 4, 5, 10
>>?)
>>6. Use an easy-to-understand, unified scoring method for all test
>>frameworks.
>>
>> As I mentioned, yesterday, Julien and Serena have been working on this,
>> as well.  Julien has developed a macro
>> <https://gerrit.opnfv.org/gerrit/#/c/48515/> to enable consolidation of
>> all results per scenario. He is intending to use the Daisy installer as a
>> platform to verify the macro, which then can be adapted to other installers.
>>
>>
>>
>> In addition, Serena has agreed to help manage an intern who can assist
>> with the project.  I have an action to c

Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

[SerenaFeng(zte)]

Hi all，

In our case, a link back to jobs on Jenkins is necessary for the latest
iteration(s, for a short period), but for the very historical records, I
don't think it is needed, after all, in very few conditions, the user needs
to check the historical log, what they care more is the
success/fail/duration/faiure-reason...etc information, which are all
already included in the database. And we have already saved
start_date/stop_date to distinguish each test results.

Considering the cost and effort of saving the logs somewhere and
permanently, VS, our requirement, I think we can still leverage build_id
for the link work, there may be some wrong leading due to the two
aforementioned reasons, but we can take it as the tolerable mistake.

And personally comments:
- recreation of jobs
   very little possibility to do it
- migration to a new jenkins instance without preserving the build history
   it is quite impossible to do like that in an official usage

What do you think? @Jack @David @Fatih @Trevor

BUILD_TIMESTAMP is something we can leverage, currently, the
start_date/stop_date is generated based on the local time, the time is
quite inconsistent due to time zone and/or ntp server issues, @Trevor and
@Fatih, do you think it can solve our mis-date issues by using
BUILD_TIMESTAMP?


BRs
Serena

On Tue, Jan 30, 2018 at 4:19 PM Fatih Degirmenci <
fatih.degirme...@ericsson.com> wrote:

> Hi Jack,
>
> As Serena noted, all installers except Apex use same job structure; 1
> parent/upstream job starting and controlling the execution of the
> child/downstream jobs that do deployment, functest, and yardstick for a
> single scenario.
> A variable can be generated by parent job and passed to child jobs so you
> can have one variable to identify a full run in the end.
>
> This could be BUILD_TIMESTAMP as Trevor talked about below or something
> else as long as it is guaranteed to be unique within that installer or
> scenario.
> What is important here is how you intend to push data to database. If you
> will track results on installer level than there is very small possibility
> of having same timestamp for 2 separate runs if they start exactly at the
> same time on millisecond.
> In this case, UUID can be generated and used instead.
>
> Another thing I missed mentioning is that you intend to link back to jobs
> on Jenkins if I understood it correctly. We have build retention policy on
> Jenkins so after certain period or no of builds (X days or Y no of builds),
> they get deleted and you will have links pointing to non-existing logs.
> If you need to keep the logs then my suggestion would be to dump logs to
> somewhere that can also use the variable generated by the parent job so you
> have references to the things correctly.
>
> /Fatih
>
> On 2018-01-30, 08:22, "opnfv-tech-discuss-boun...@lists.opnfv.org on
> behalf of chenjiankun" <opnfv-tech-discuss-boun...@lists.opnfv.org on
> behalf of chenjiank...@huawei.com> wrote:
>
> Hi Fatih & Trevor,
>
> What we need is a new variable(like build_id) which can identify a
> combination of (Deployment + Functest + Yardstick) CI jobs. For example, if
> in one curtain scenario, we will run deployment job, Functest job and
> Yardstick job, and then their id will be the same.
> Since I am not familiar with the Jenkins, can you give us some
> suggestions?
> Or can you help us to add a new variable which can satisfy our
> requirements?
> Thanks in advance~
>
> BRs,
> Jack Chan
>
> -邮件原件-
> 发件人: opnfv-tech-discuss-boun...@lists.opnfv.org [mailto:
> opnfv-tech-discuss-boun...@lists.opnfv.org] 代表 Trevor Bramwell
> 发送时间: 2018年1月30日 7:21
> 收件人: Fatih Degirmenci
> 抄送: Serena Feng; TECH-DISCUSS OPNFV
> 主题: Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources
> collection
>
> Hi,
>
> Previous versions of Jenkins set BUILD_ID to the timestamp of the
> build but now BUILD_ID = BUILD_NUMBER.
>
> There are two plugins that appear to bring back
> 'BUILD_TIMESTAMP'[1][2].
>
> It also wouldn't be too hard to generate this as part of a build step
> and we could keep the format TIMESTAMP_FORMAT as a global variable.
>
> Regards,
> Trevor Bramwell
>
> [1] https://plugins.jenkins.io/build-timestamp
> [2] https://plugins.jenkins.io/zentimestamp
>
> On Tue, Jan 30, 2018 at 12:04:10AM +0100, Fatih Degirmenci wrote:
> > Hi,
> >
> > BUILD_ID should not be used for this type of purposes since it gets
> > reset when certain things happen such as
> >
> > - recreation of jobs
> > - migration to a new jenkins instance without preserving the build
> > history
> >
>

Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

[Fatih Degirmenci]

Hi Jack,

As Serena noted, all installers except Apex use same job structure; 1 
parent/upstream job starting and controlling the execution of the 
child/downstream jobs that do deployment, functest, and yardstick for a single 
scenario.
A variable can be generated by parent job and passed to child jobs so you can 
have one variable to identify a full run in the end.

This could be BUILD_TIMESTAMP as Trevor talked about below or something else as 
long as it is guaranteed to be unique within that installer or scenario.
What is important here is how you intend to push data to database. If you will 
track results on installer level than there is very small possibility of having 
same timestamp for 2 separate runs if they start exactly at the same time on 
millisecond.
In this case, UUID can be generated and used instead.

Another thing I missed mentioning is that you intend to link back to jobs on 
Jenkins if I understood it correctly. We have build retention policy on Jenkins 
so after certain period or no of builds (X days or Y no of builds), they get 
deleted and you will have links pointing to non-existing logs.
If you need to keep the logs then my suggestion would be to dump logs to 
somewhere that can also use the variable generated by the parent job so you 
have references to the things correctly.

/Fatih
 
On 2018-01-30, 08:22, "opnfv-tech-discuss-boun...@lists.opnfv.org on behalf of 
chenjiankun" <opnfv-tech-discuss-boun...@lists.opnfv.org on behalf of 
chenjiank...@huawei.com> wrote:

Hi Fatih & Trevor,

What we need is a new variable(like build_id) which can identify a 
combination of (Deployment + Functest + Yardstick) CI jobs. For example, if in 
one curtain scenario, we will run deployment job, Functest job and Yardstick 
job, and then their id will be the same.
Since I am not familiar with the Jenkins, can you give us some suggestions?
Or can you help us to add a new variable which can satisfy our requirements?
Thanks in advance~ 

BRs,
Jack Chan

-邮件原件-
发件人: opnfv-tech-discuss-boun...@lists.opnfv.org 
[mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] 代表 Trevor Bramwell
发送时间: 2018年1月30日 7:21
收件人: Fatih Degirmenci
抄送: Serena Feng; TECH-DISCUSS OPNFV
    主题: Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources 
collection

Hi,

Previous versions of Jenkins set BUILD_ID to the timestamp of the build but 
now BUILD_ID = BUILD_NUMBER.

There are two plugins that appear to bring back 'BUILD_TIMESTAMP'[1][2].

It also wouldn't be too hard to generate this as part of a build step and 
we could keep the format TIMESTAMP_FORMAT as a global variable.

Regards,
Trevor Bramwell

[1] https://plugins.jenkins.io/build-timestamp
[2] https://plugins.jenkins.io/zentimestamp

On Tue, Jan 30, 2018 at 12:04:10AM +0100, Fatih Degirmenci wrote:
> Hi,
> 
> BUILD_ID should not be used for this type of purposes since it gets 
> reset when certain things happen such as
> 
> - recreation of jobs
> - migration to a new jenkins instance without preserving the build 
> history
> 
> If any of these things happen, there might be results with the same id 
> in db over time. It is better to find something else that never gets 
> reset such as timestamp.
> 
> /Fatih
> 
> On 29 Jan 2018, at 07:39, SerenaFeng(zte) 
> <serena.feng.711+...@gmail.com>
> wrote:
> 
> Hi Jack
> 
> Eventually, we hope all the installers push their deployment results 
> to TestAPI, in the current stage, we just leverage Daisy to have a try 
> and find a common framework to achieve that, we will promote it to 
> other installers soon after it works.
> 
> In terms of Jenkins id, almost all the installers leverage muti-job to 
> manage deployment and drive test projects. there is an entry job to 
> control all the jobs, such as 
> daisy-os-nosdn-nofeature-ha-baremetal-daily-master
> <https://build.opnfv.org/ci/job/daisy-os-nosdn-nofeature-ha-baremetal-
> daily-master/>, maybe we can leverage the combination of the Jenkins 
> job and build_id
> daisy-os-nosdn-nofeature-ha-baremetal-daily-master_185
> <https://build.opnfv.org/ci/job/daisy-os-nosdn-nofeature-ha-baremetal-
> daily-master/185/>(or jest the Jenkins job) to link to the Jenkins. As 
> for naming, instead of 'Jenkins' I would recommend using 'Job_name', 
> because, in zuulv3, Jenkins will be discarded, but the job idea will 
> be left(maybe in some other way)
> 
> For the status show, if only show the last and latest *one* iteration, 
> simple pass/fail is fine, or else multi-results will be included, I 
> don't rec

Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

[chenjiankun]

Hi Julien,

Sorry for missing the email you have cc to me and the meeting.

Can you update the relevant information here then?

Thanks in advance. ☺

BRs,
Jack Chan

发件人: Julien [mailto:julien...@gmail.com]
发送时间: 2018年1月27日 23:42
收件人: David McBride
抄送: chenjiankun; SerenaFeng(zte); Serena Feng; TECH-DISCUSS OPNFV
主题: Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

Hi Jack,

I have finished the macro, but it failed for a Jenkins plugin is missing. I cc 
the issue email to you, which will be discussed in the infra meeting.

BR/Julien

David McBride 
<dmcbr...@linuxfoundation.org<mailto:dmcbr...@linuxfoundation.org>>于2018年1月27日周六
 上午12:13写道：
+Julien

Jack,

I like the table format and the filtering options that you proposed.  Also +1 
to Serena's suggestion to break out the health check data.

I think that Julien is working on a Jenkins macro to enable the test API to 
retrieve the "jenkins id".  Julien - please confirm.

David

On Tue, Jan 23, 2018 at 10:53 PM, chenjiankun 
<chenjiank...@huawei.com<mailto:chenjiank...@huawei.com>> wrote:
Hi Serena,

Thanks for your information updating.
I am so glad to see TestAPI is ready to collecting deployment results, and it 
would be much more convenient.
One question: does all installer will push result to DB, or just Daisy for now?

For the Jenkins id, it means a combination of deployment + functest + yardstick 
job. I am not sure if there are a ‘jenkins_id’ for now. This field is for 
identify the combination jobs.
For the second comment, maybe there are some misunderstanding, as David’s 
requirements, we will show data last x iterations(not days), so I think there 
will be no such problems.
For the healthcheck column, I totally agree with you. But maybe it will depend 
on functest to upload test result independently.

What’s your opinions? @David, @Serena

BRs,
Jack Chan

发件人: SerenaFeng(zte) 
[mailto:serena.feng.711+...@gmail.com<mailto:serena.feng.711%2b...@gmail.com>]
发送时间: 2018年1月22日 18:08
收件人: chenjiankun
抄送: David McBride; Serena Feng; TECH-DISCUSS OPNFV
主题: Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

Hi David & Jack,

Thanks for mentioning the task, an interface for collecting deployment results 
has been ready in TestAPI[1],
the Macro for pushing deployment result is also ready by Jelien[2], I believe 
it will facilitate all the installers' work,
currently, Julien is working on pushing daisy results to TestAPI leveraging 
that Macro, I think it will be finished soon.

And regarding how to show all the information in a table, I suggest we can take 
a look at Jack's proposal first.

@Jack, a few comments:
1. what's the opinion of jenkins id?
2. for a scenario-installer combination, some will not run once in a 
day(trigger multiple times or run in multiple pods),
in this case, a simple pass/fail will be too vague, and to facilitate the 
support of data iteration, I would suggest
leveraging 8/9/10(8 passed, 9 triggered, 10total), delete the final 
statistic line
3. how about adding a healthcheck column(functest-healthcheck test cases), to 
see if the installer meets the milestone 3.0


[1]: https://gerrit.opnfv.org/gerrit/#/c/49895/
[2]: https://gerrit.opnfv.org/gerrit/#/c/48515/

BRs
Serena

On Mon, Jan 22, 2018 at 5:02 PM chenjiankun 
<chenjiank...@huawei.com<mailto:chenjiank...@huawei.com>> wrote:
Thanks, David.

According your descriptions, I have created a demo table as below(wish I do not 
misunderstanding your meaning):

scenario

date

Jenkins

Version

Installer

Deployment

Functest

yardstick

os-nosdn-nofeature-ha

2018-01-21

Jenkins id

euphrates

compass

pass

pass

pass


2018-01-21

Jenkins id

euphrates

compass

fail

not trigger

not trigger


statistic

8/9/10
(pass:8,triggered:9, total:10)

6/7/8

6/7/8



This last line in table body is the statistics information, and lines above are 
the detailed information(and it can be folded).
The score 8/9/10 is pass/triggered/total. Total means should run, triggered 
means actually run.
Also we can add three filters:

If you select installer as compass, then will show all data related to compass.
Iterations means last x data points to be displayed.

Does this table satisfied your requirements?

BRs,
Jack Chan
发件人: David McBride 
[mailto:dmcbr...@linuxfoundation.org<mailto:dmcbr...@linuxfoundation.org>]
发送时间: 2018年1月20日 3:07
收件人: chenjiankun
抄送: TECH-DISCUSS OPNFV; tro...@redhat.com<mailto:tro...@redhat.com>; Brattain, 
Ross B; Rao, Sridhar; OLLIVIER Cédric IMT/OLN; 
mark.bei...@dell.com<mailto:mark.bei...@dell.com>; Yuyang (Gabriel); ALFRED C 
'MORTON ' (acmor...@att.com<mailto:acmor...@att.com>); 
emma.l.fo...@intel.com<mailto:emma.l.fo...@intel.com>; Liyin (Ace); Wangwulin 
(Linda); georg.k...@ericsson.com<mailto:georg.k...@ericsson.com>; Serena Feng; 
Julien
主题: Re: [opnfv-tech-discuss][test-wg]Requirements for test resources col

Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

[chenjiankun]

Hi Serena,

For the Jenkins id, as Fatih and Trevor clarification, we can’t use build_id to 
implement. So maybe we can discuss with them in detail and try to add a new 
variable in Jenkins. As for the name, Fatih and Trevor maybe are  familiar with 
the Jenkins, so maybe they can offer a more accuracy name after we clarify our 
requirements.
As for the status, both are OK to me. Since I get this requirements from David, 
in the previous email he said “Display the data above for the last x data 
points (e.g., 4, 5, 10 ?) “, and I add a statistic line to show more 
information. So for this issue, maybe it’s better to leave it to David to 
clarify what information he want to display here.
@David, can you give us some clarification about what information you want to 
see here?
For the health check, Linda told me that maybe later Functest will upload 
health check status independently. If so, it would be easy to get health result 
then. For now we can combine the connection_check/api_check/_snaps_health_check 
result to get a total status.

BRs,
Jack Chan

发件人: SerenaFeng(zte) [mailto:serena.feng.711+...@gmail.com]
发送时间: 2018年1月29日 14:40
收件人: Julien
抄送: David McBride; chenjiankun; Serena Feng; TECH-DISCUSS OPNFV
主题: Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

Hi Jack

Eventually, we hope all the installers push their deployment results to 
TestAPI, in the current stage, we just leverage Daisy to have a try and find a 
common framework to achieve that, we will promote it to other installers soon 
after it works.

In terms of Jenkins id, almost all the installers leverage muti-job to manage 
deployment and drive test projects. there is an entry job to control all the 
jobs, such as 
daisy-os-nosdn-nofeature-ha-baremetal-daily-master<https://build.opnfv.org/ci/job/daisy-os-nosdn-nofeature-ha-baremetal-daily-master/>,
 maybe we can leverage the combination of the Jenkins job and build_id 
daisy-os-nosdn-nofeature-ha-baremetal-daily-master_185<https://build.opnfv.org/ci/job/daisy-os-nosdn-nofeature-ha-baremetal-daily-master/185/>(or
 jest the Jenkins job) to link to the Jenkins. As for naming, instead of 
'Jenkins' I would recommend using 'Job_name', because, in zuulv3, Jenkins will 
be discarded, but the job idea will be left(maybe in some other way)

For the status show, if only show the last and latest one iteration, simple 
pass/fail is fine, or else multi-results will be included, I don't recommend 
that. Personally, my suggestion is either integrate last one iteration into 
last x iterations, all use statistics to show the results, or separate into two 
pages: the current status(simple pass/fail) and the history statistic 
status(8/9/10).

For Functest health check, Sure, Functest reports per execution result per test 
case separately, we can get them from TestAPI, there's no problem with that. 
But for health check actually includes 3 test 
cases(connection_check/api_check/snaps_health_check), we need to cope with that 
carefully.


BRs
Serena

On Sat, Jan 27, 2018 at 11:42 PM Julien 
<julien...@gmail.com<mailto:julien...@gmail.com>> wrote:
Hi Jack,

I have finished the macro, but it failed for a Jenkins plugin is missing. I cc 
the issue email to you, which will be discussed in the infra meeting.

BR/Julien

David McBride 
<dmcbr...@linuxfoundation.org<mailto:dmcbr...@linuxfoundation.org>>于2018年1月27日周六
 上午12:13写道：
+Julien

Jack,

I like the table format and the filtering options that you proposed.  Also +1 
to Serena's suggestion to break out the health check data.

I think that Julien is working on a Jenkins macro to enable the test API to 
retrieve the "jenkins id".  Julien - please confirm.

David

On Tue, Jan 23, 2018 at 10:53 PM, chenjiankun 
<chenjiank...@huawei.com<mailto:chenjiank...@huawei.com>> wrote:
Hi Serena,

Thanks for your information updating.
I am so glad to see TestAPI is ready to collecting deployment results, and it 
would be much more convenient.
One question: does all installer will push result to DB, or just Daisy for now?

For the Jenkins id, it means a combination of deployment + functest + yardstick 
job. I am not sure if there are a ‘jenkins_id’ for now. This field is for 
identify the combination jobs.
For the second comment, maybe there are some misunderstanding, as David’s 
requirements, we will show data last x iterations(not days), so I think there 
will be no such problems.
For the healthcheck column, I totally agree with you. But maybe it will depend 
on functest to upload test result independently.

What’s your opinions? @David, @Serena

BRs,
Jack Chan

发件人: SerenaFeng(zte) 
[mailto:serena.feng.711+...@gmail.com<mailto:serena.feng.711%2b...@gmail.com>]
发送时间: 2018年1月22日 18:08
收件人: chenjiankun
抄送: David McBride; Serena Feng; TECH-DISCUSS OPNFV
主题: Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

Hi David & Jack,

Thanks for mentioning the 

Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

[Trevor Bramwell]

Hi,

Previous versions of Jenkins set BUILD_ID to the timestamp of the build
but now BUILD_ID = BUILD_NUMBER.

There are two plugins that appear to bring back 'BUILD_TIMESTAMP'[1][2].

It also wouldn't be too hard to generate this as part of a build step
and we could keep the format TIMESTAMP_FORMAT as a global variable.

Regards,
Trevor Bramwell

[1] https://plugins.jenkins.io/build-timestamp
[2] https://plugins.jenkins.io/zentimestamp

On Tue, Jan 30, 2018 at 12:04:10AM +0100, Fatih Degirmenci wrote:
> Hi,
> 
> BUILD_ID should not be used for this type of purposes since it gets reset
> when certain things happen such as
> 
> - recreation of jobs
> - migration to a new jenkins instance without preserving the build history
> 
> If any of these things happen, there might be results with the same id in
> db over time. It is better to find something else that never gets reset
> such as timestamp.
> 
> /Fatih
> 
> On 29 Jan 2018, at 07:39, SerenaFeng(zte) <serena.feng.711+...@gmail.com>
> wrote:
> 
> Hi Jack
> 
> Eventually, we hope all the installers push their deployment results to
> TestAPI, in the current stage, we just leverage Daisy to have a try and
> find a common framework to achieve that, we will promote it to other
> installers soon after it works.
> 
> In terms of Jenkins id, almost all the installers leverage muti-job to
> manage deployment and drive test projects. there is an entry job to control
> all the jobs, such as daisy-os-nosdn-nofeature-ha-baremetal-daily-master
> <https://build.opnfv.org/ci/job/daisy-os-nosdn-nofeature-ha-baremetal-daily-master/>,
> maybe we can leverage the combination of the Jenkins job and build_id
> daisy-os-nosdn-nofeature-ha-baremetal-daily-master_185
> <https://build.opnfv.org/ci/job/daisy-os-nosdn-nofeature-ha-baremetal-daily-master/185/>(or
> jest the Jenkins job) to link to the Jenkins. As for naming, instead of
> 'Jenkins' I would recommend using 'Job_name', because, in zuulv3, Jenkins
> will be discarded, but the job idea will be left(maybe in some other way)
> 
> For the status show, if only show the last and latest *one* iteration,
> simple pass/fail is fine, or else multi-results will be included, I don't
> recommend that. Personally, my suggestion is either integrate last one
> iteration into last x iterations, all use statistics to show the results,
> or separate into two pages: the current status(simple pass/fail) and the
> history statistic status(8/9/10).
> 
> For Functest health check, Sure, Functest reports per execution result per
> test case separately, we can get them from TestAPI, there's no problem with
> that. But for health check actually includes 3 test
> cases(connection_check/api_check/snaps_health_check), we need to cope with
> that carefully.
> 
> 
> BRs
> Serena
> 
> On Sat, Jan 27, 2018 at 11:42 PM Julien <julien...@gmail.com> wrote:
> 
> > Hi Jack,
> >
> > I have finished the macro, but it failed for a Jenkins plugin is missing.
> > I cc the issue email to you, which will be discussed in the infra meeting.
> >
> > BR/Julien
> >
> > David McBride <dmcbr...@linuxfoundation.org>于2018年1月27日周六 上午12:13写道：
> >
> >> +Julien
> >>
> >> Jack,
> >>
> >> I like the table format and the filtering options that you proposed.
> >> Also +1 to Serena's suggestion to break out the health check data.
> >>
> >> I think that Julien is working on a Jenkins macro to enable the test API
> >> to retrieve the "jenkins id".  Julien - please confirm.
> >>
> >> David
> >>
> >> On Tue, Jan 23, 2018 at 10:53 PM, chenjiankun <chenjiank...@huawei.com>
> >> wrote:
> >>
> >>> Hi Serena,
> >>>
> >>>
> >>>
> >>> Thanks for your information updating.
> >>>
> >>> I am so glad to see TestAPI is ready to collecting deployment results,
> >>> and it would be much more convenient.
> >>>
> >>> One question: does all installer will push result to DB, or just Daisy
> >>> for now?
> >>>
> >>>
> >>>
> >>> For the Jenkins id, it means a combination of deployment + functest +
> >>> yardstick job. I am not sure if there are a ‘jenkins_id’ for now. This
> >>> field is for identify the combination jobs.
> >>>
> >>> For the second comment, maybe there are some misunderstanding, as
> >>> David’s requirements, we will show data last x iterations(not days), so I
> >>> think there will be no such problems.
> >>>
> >>> For t

Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

[Fatih Degirmenci]

Hi,

BUILD_ID should not be used for this type of purposes since it gets reset
when certain things happen such as

- recreation of jobs
- migration to a new jenkins instance without preserving the build history

If any of these things happen, there might be results with the same id in
db over time. It is better to find something else that never gets reset
such as timestamp.

/Fatih

On 29 Jan 2018, at 07:39, SerenaFeng(zte) <serena.feng.711+...@gmail.com>
wrote:

Hi Jack

Eventually, we hope all the installers push their deployment results to
TestAPI, in the current stage, we just leverage Daisy to have a try and
find a common framework to achieve that, we will promote it to other
installers soon after it works.

In terms of Jenkins id, almost all the installers leverage muti-job to
manage deployment and drive test projects. there is an entry job to control
all the jobs, such as daisy-os-nosdn-nofeature-ha-baremetal-daily-master
<https://build.opnfv.org/ci/job/daisy-os-nosdn-nofeature-ha-baremetal-daily-master/>,
maybe we can leverage the combination of the Jenkins job and build_id
daisy-os-nosdn-nofeature-ha-baremetal-daily-master_185
<https://build.opnfv.org/ci/job/daisy-os-nosdn-nofeature-ha-baremetal-daily-master/185/>(or
jest the Jenkins job) to link to the Jenkins. As for naming, instead of
'Jenkins' I would recommend using 'Job_name', because, in zuulv3, Jenkins
will be discarded, but the job idea will be left(maybe in some other way)

For the status show, if only show the last and latest *one* iteration,
simple pass/fail is fine, or else multi-results will be included, I don't
recommend that. Personally, my suggestion is either integrate last one
iteration into last x iterations, all use statistics to show the results,
or separate into two pages: the current status(simple pass/fail) and the
history statistic status(8/9/10).

For Functest health check, Sure, Functest reports per execution result per
test case separately, we can get them from TestAPI, there's no problem with
that. But for health check actually includes 3 test
cases(connection_check/api_check/snaps_health_check), we need to cope with
that carefully.


BRs
Serena

On Sat, Jan 27, 2018 at 11:42 PM Julien <julien...@gmail.com> wrote:

> Hi Jack,
>
> I have finished the macro, but it failed for a Jenkins plugin is missing.
> I cc the issue email to you, which will be discussed in the infra meeting.
>
> BR/Julien
>
> David McBride <dmcbr...@linuxfoundation.org>于2018年1月27日周六 上午12:13写道：
>
>> +Julien
>>
>> Jack,
>>
>> I like the table format and the filtering options that you proposed.
>> Also +1 to Serena's suggestion to break out the health check data.
>>
>> I think that Julien is working on a Jenkins macro to enable the test API
>> to retrieve the "jenkins id".  Julien - please confirm.
>>
>> David
>>
>> On Tue, Jan 23, 2018 at 10:53 PM, chenjiankun <chenjiank...@huawei.com>
>> wrote:
>>
>>> Hi Serena,
>>>
>>>
>>>
>>> Thanks for your information updating.
>>>
>>> I am so glad to see TestAPI is ready to collecting deployment results,
>>> and it would be much more convenient.
>>>
>>> One question: does all installer will push result to DB, or just Daisy
>>> for now?
>>>
>>>
>>>
>>> For the Jenkins id, it means a combination of deployment + functest +
>>> yardstick job. I am not sure if there are a ‘jenkins_id’ for now. This
>>> field is for identify the combination jobs.
>>>
>>> For the second comment, maybe there are some misunderstanding, as
>>> David’s requirements, we will show data last x iterations(not days), so I
>>> think there will be no such problems.
>>>
>>> For the healthcheck column, I totally agree with you. But maybe it will
>>> depend on functest to upload test result independently.
>>>
>>>
>>>
>>> What’s your opinions? @David, @Serena
>>>
>>>
>>>
>>> BRs,
>>>
>>> Jack Chan
>>>
>>>
>>>
>>> *发件人:* SerenaFeng(zte) [mailto:serena.feng.711+...@gmail.com]
>>> *发送时间:* 2018年1月22日 18:08
>>> *收件人:* chenjiankun
>>> *抄送:* David McBride; Serena Feng; TECH-DISCUSS OPNFV
>>> *主题:* Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources
>>> collection
>>>
>>>
>>>
>>> Hi David & Jack,
>>>
>>>
>>>
>>> Thanks for mentioning the task, an interface for collecting deployment
>>> results has been ready in TestAPI[1],
>>>
>>> the Macro for pushing deployment result is also ready by Jelien[

Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

[SerenaFeng(zte)]

Hi Jack

Eventually, we hope all the installers push their deployment results to
TestAPI, in the current stage, we just leverage Daisy to have a try and
find a common framework to achieve that, we will promote it to other
installers soon after it works.

In terms of Jenkins id, almost all the installers leverage muti-job to
manage deployment and drive test projects. there is an entry job to control
all the jobs, such as daisy-os-nosdn-nofeature-ha-baremetal-daily-master
<https://build.opnfv.org/ci/job/daisy-os-nosdn-nofeature-ha-baremetal-daily-master/>,
maybe we can leverage the combination of the Jenkins job and build_id
daisy-os-nosdn-nofeature-ha-baremetal-daily-master_185
<https://build.opnfv.org/ci/job/daisy-os-nosdn-nofeature-ha-baremetal-daily-master/185/>(or
jest the Jenkins job) to link to the Jenkins. As for naming, instead of
'Jenkins' I would recommend using 'Job_name', because, in zuulv3, Jenkins
will be discarded, but the job idea will be left(maybe in some other way)

For the status show, if only show the last and latest *one* iteration,
simple pass/fail is fine, or else multi-results will be included, I don't
recommend that. Personally, my suggestion is either integrate last one
iteration into last x iterations, all use statistics to show the results,
or separate into two pages: the current status(simple pass/fail) and the
history statistic status(8/9/10).

For Functest health check, Sure, Functest reports per execution result per
test case separately, we can get them from TestAPI, there's no problem with
that. But for health check actually includes 3 test
cases(connection_check/api_check/snaps_health_check), we need to cope with
that carefully.


BRs
Serena

On Sat, Jan 27, 2018 at 11:42 PM Julien <julien...@gmail.com> wrote:

> Hi Jack,
>
> I have finished the macro, but it failed for a Jenkins plugin is missing.
> I cc the issue email to you, which will be discussed in the infra meeting.
>
> BR/Julien
>
> David McBride <dmcbr...@linuxfoundation.org>于2018年1月27日周六 上午12:13写道：
>
>> +Julien
>>
>> Jack,
>>
>> I like the table format and the filtering options that you proposed.
>> Also +1 to Serena's suggestion to break out the health check data.
>>
>> I think that Julien is working on a Jenkins macro to enable the test API
>> to retrieve the "jenkins id".  Julien - please confirm.
>>
>> David
>>
>> On Tue, Jan 23, 2018 at 10:53 PM, chenjiankun <chenjiank...@huawei.com>
>> wrote:
>>
>>> Hi Serena,
>>>
>>>
>>>
>>> Thanks for your information updating.
>>>
>>> I am so glad to see TestAPI is ready to collecting deployment results,
>>> and it would be much more convenient.
>>>
>>> One question: does all installer will push result to DB, or just Daisy
>>> for now?
>>>
>>>
>>>
>>> For the Jenkins id, it means a combination of deployment + functest +
>>> yardstick job. I am not sure if there are a ‘jenkins_id’ for now. This
>>> field is for identify the combination jobs.
>>>
>>> For the second comment, maybe there are some misunderstanding, as
>>> David’s requirements, we will show data last x iterations(not days), so I
>>> think there will be no such problems.
>>>
>>> For the healthcheck column, I totally agree with you. But maybe it will
>>> depend on functest to upload test result independently.
>>>
>>>
>>>
>>> What’s your opinions? @David, @Serena
>>>
>>>
>>>
>>> BRs,
>>>
>>> Jack Chan
>>>
>>>
>>>
>>> *发件人:* SerenaFeng(zte) [mailto:serena.feng.711+...@gmail.com]
>>> *发送时间:* 2018年1月22日 18:08
>>> *收件人:* chenjiankun
>>> *抄送:* David McBride; Serena Feng; TECH-DISCUSS OPNFV
>>> *主题:* Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources
>>> collection
>>>
>>>
>>>
>>> Hi David & Jack,
>>>
>>>
>>>
>>> Thanks for mentioning the task, an interface for collecting deployment
>>> results has been ready in TestAPI[1],
>>>
>>> the Macro for pushing deployment result is also ready by Jelien[2], I
>>> believe it will facilitate all the installers' work,
>>>
>>> currently, Julien is working on pushing daisy results to TestAPI
>>> leveraging that Macro, I think it will be finished soon.
>>>
>>>
>>>
>>> And regarding how to show all the information in a table, I suggest we
>>> can take a look at Jack's proposal first.
>>>
>>>
>>>
>>> @Jack, a few co

Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

[Julien]

Hi Jack,

I have finished the macro, but it failed for a Jenkins plugin is missing. I
cc the issue email to you, which will be discussed in the infra meeting.

BR/Julien

David McBride <dmcbr...@linuxfoundation.org>于2018年1月27日周六 上午12:13写道：

> +Julien
>
> Jack,
>
> I like the table format and the filtering options that you proposed.  Also
> +1 to Serena's suggestion to break out the health check data.
>
> I think that Julien is working on a Jenkins macro to enable the test API
> to retrieve the "jenkins id".  Julien - please confirm.
>
> David
>
> On Tue, Jan 23, 2018 at 10:53 PM, chenjiankun <chenjiank...@huawei.com>
> wrote:
>
>> Hi Serena,
>>
>>
>>
>> Thanks for your information updating.
>>
>> I am so glad to see TestAPI is ready to collecting deployment results,
>> and it would be much more convenient.
>>
>> One question: does all installer will push result to DB, or just Daisy
>> for now?
>>
>>
>>
>> For the Jenkins id, it means a combination of deployment + functest +
>> yardstick job. I am not sure if there are a ‘jenkins_id’ for now. This
>> field is for identify the combination jobs.
>>
>> For the second comment, maybe there are some misunderstanding, as David’s
>> requirements, we will show data last x iterations(not days), so I think
>> there will be no such problems.
>>
>> For the healthcheck column, I totally agree with you. But maybe it will
>> depend on functest to upload test result independently.
>>
>>
>>
>> What’s your opinions? @David, @Serena
>>
>>
>>
>> BRs,
>>
>> Jack Chan
>>
>>
>>
>> *发件人:* SerenaFeng(zte) [mailto:serena.feng.711+...@gmail.com]
>> *发送时间:* 2018年1月22日 18:08
>> *收件人:* chenjiankun
>> *抄送:* David McBride; Serena Feng; TECH-DISCUSS OPNFV
>> *主题:* Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources
>> collection
>>
>>
>>
>> Hi David & Jack,
>>
>>
>>
>> Thanks for mentioning the task, an interface for collecting deployment
>> results has been ready in TestAPI[1],
>>
>> the Macro for pushing deployment result is also ready by Jelien[2], I
>> believe it will facilitate all the installers' work,
>>
>> currently, Julien is working on pushing daisy results to TestAPI
>> leveraging that Macro, I think it will be finished soon.
>>
>>
>>
>> And regarding how to show all the information in a table, I suggest we
>> can take a look at Jack's proposal first.
>>
>>
>>
>> @Jack, a few comments:
>>
>> 1. what's the opinion of jenkins id?
>>
>> 2. for a scenario-installer combination, some will not run once in a
>> day(trigger multiple times or run in multiple pods),
>>
>> in this case, a simple pass/fail will be too vague, and to facilitate
>> the support of data iteration, I would suggest
>>
>> leveraging 8/9/10(8 passed, 9 triggered, 10total), delete the
>> final statistic line
>>
>> 3. how about adding a healthcheck column(functest-healthcheck test
>> cases), to see if the installer meets the milestone 3.0
>>
>>
>>
>>
>>
>> [1]: https://gerrit.opnfv.org/gerrit/#/c/49895/
>>
>> [2]: https://gerrit.opnfv.org/gerrit/#/c/48515/
>>
>>
>>
>> BRs
>>
>> Serena
>>
>>
>>
>> On Mon, Jan 22, 2018 at 5:02 PM chenjiankun <chenjiank...@huawei.com>
>> wrote:
>>
>> Thanks, David.
>>
>>
>>
>> According your descriptions, I have created a demo table as below(wish I
>> do not misunderstanding your meaning):
>>
>>
>>
>> *scenario *
>>
>> *date *
>>
>> *Jenkins *
>>
>> *Version *
>>
>> *Installer *
>>
>> *Deployment *
>>
>> *Functest *
>>
>> *yardstick *
>>
>> os-nosdn-nofeature-ha
>>
>> 2018-01-21
>>
>> Jenkins id
>>
>> euphrates
>>
>> compass
>>
>> pass
>>
>> pass
>>
>> pass
>>
>> 2018-01-21
>>
>> Jenkins id
>>
>> euphrates
>>
>> compass
>>
>> fail
>>
>> not trigger
>>
>> not trigger
>>
>> statistic
>>
>> 8/9/10
>>
>> (pass:8,triggered:9, total:10)
>>
>> 6/7/8
>>
>> 6/7/8
>>
>>
>>
>>
>>
>> This last line in table body is the statistics information, and lines
>> above are the detailed info

Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

[David McBride]

+Julien

Jack,

I like the table format and the filtering options that you proposed.  Also
+1 to Serena's suggestion to break out the health check data.

I think that Julien is working on a Jenkins macro to enable the test API to
retrieve the "jenkins id".  Julien - please confirm.

David

On Tue, Jan 23, 2018 at 10:53 PM, chenjiankun <chenjiank...@huawei.com>
wrote:

> Hi Serena,
>
>
>
> Thanks for your information updating.
>
> I am so glad to see TestAPI is ready to collecting deployment results, and
> it would be much more convenient.
>
> One question: does all installer will push result to DB, or just Daisy for
> now?
>
>
>
> For the Jenkins id, it means a combination of deployment + functest +
> yardstick job. I am not sure if there are a ‘jenkins_id’ for now. This
> field is for identify the combination jobs.
>
> For the second comment, maybe there are some misunderstanding, as David’s
> requirements, we will show data last x iterations(not days), so I think
> there will be no such problems.
>
> For the healthcheck column, I totally agree with you. But maybe it will
> depend on functest to upload test result independently.
>
>
>
> What’s your opinions? @David, @Serena
>
>
>
> BRs,
>
> Jack Chan
>
>
>
> *发件人:* SerenaFeng(zte) [mailto:serena.feng.711+...@gmail.com]
> *发送时间:* 2018年1月22日 18:08
> *收件人:* chenjiankun
> *抄送:* David McBride; Serena Feng; TECH-DISCUSS OPNFV
> *主题:* Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources
> collection
>
>
>
> Hi David & Jack,
>
>
>
> Thanks for mentioning the task, an interface for collecting deployment
> results has been ready in TestAPI[1],
>
> the Macro for pushing deployment result is also ready by Jelien[2], I
> believe it will facilitate all the installers' work,
>
> currently, Julien is working on pushing daisy results to TestAPI
> leveraging that Macro, I think it will be finished soon.
>
>
>
> And regarding how to show all the information in a table, I suggest we can
> take a look at Jack's proposal first.
>
>
>
> @Jack, a few comments:
>
> 1. what's the opinion of jenkins id?
>
> 2. for a scenario-installer combination, some will not run once in a
> day(trigger multiple times or run in multiple pods),
>
> in this case, a simple pass/fail will be too vague, and to facilitate
> the support of data iteration, I would suggest
>
> leveraging 8/9/10(8 passed, 9 triggered, 10total), delete the
> final statistic line
>
> 3. how about adding a healthcheck column(functest-healthcheck test cases),
> to see if the installer meets the milestone 3.0
>
>
>
>
>
> [1]: https://gerrit.opnfv.org/gerrit/#/c/49895/
>
> [2]: https://gerrit.opnfv.org/gerrit/#/c/48515/
>
>
>
> BRs
>
> Serena
>
>
>
> On Mon, Jan 22, 2018 at 5:02 PM chenjiankun <chenjiank...@huawei.com>
> wrote:
>
> Thanks, David.
>
>
>
> According your descriptions, I have created a demo table as below(wish I
> do not misunderstanding your meaning):
>
>
>
> *scenario *
>
> *date *
>
> *Jenkins *
>
> *Version *
>
> *Installer *
>
> *Deployment *
>
> *Functest *
>
> *yardstick *
>
> os-nosdn-nofeature-ha
>
> 2018-01-21
>
> Jenkins id
>
> euphrates
>
> compass
>
> pass
>
> pass
>
> pass
>
> 2018-01-21
>
> Jenkins id
>
> euphrates
>
> compass
>
> fail
>
> not trigger
>
> not trigger
>
> statistic
>
> 8/9/10
>
> (pass:8,triggered:9, total:10)
>
> 6/7/8
>
> 6/7/8
>
>
>
>
>
> This last line in table body is the statistics information, and lines
> above are the detailed information(and it can be folded).
>
> The score 8/9/10 is pass/triggered/total. Total means should run,
> triggered means actually run.
>
> Also we can add three filters:
>
>
>
> If you select installer as compass, then will show all data related to
> compass.
>
> Iterations means last x data points to be displayed.
>
>
>
> Does this table satisfied your requirements?
>
>
>
> BRs,
>
> Jack Chan
>
> *发件人:* David McBride [mailto:dmcbr...@linuxfoundation.org]
> *发送时间:* 2018年1月20日 3:07
> *收件人:* chenjiankun
> *抄送:* TECH-DISCUSS OPNFV; tro...@redhat.com; Brattain, Ross B; Rao,
> Sridhar; OLLIVIER Cédric IMT/OLN; mark.bei...@dell.com; Yuyang (Gabriel);
> ALFRED C 'MORTON ' (acmor...@att.com); emma.l.fo...@intel.com; Liyin
> (Ace); Wangwulin (Linda); georg.k...@ericsson.com; Serena Feng; Julien
> *主题:* Re: [opnfv-tech-discuss][test-wg]Requirements for test resources
> collection
>
>
>
> +Serena, Julie

Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

[chenjiankun]

Hi Serena,

Thanks for your information updating.
I am so glad to see TestAPI is ready to collecting deployment results, and it 
would be much more convenient.
One question: does all installer will push result to DB, or just Daisy for now?

For the Jenkins id, it means a combination of deployment + functest + yardstick 
job. I am not sure if there are a ‘jenkins_id’ for now. This field is for 
identify the combination jobs.
For the second comment, maybe there are some misunderstanding, as David’s 
requirements, we will show data last x iterations(not days), so I think there 
will be no such problems.
For the healthcheck column, I totally agree with you. But maybe it will depend 
on functest to upload test result independently.

What’s your opinions? @David, @Serena

BRs,
Jack Chan

发件人: SerenaFeng(zte) [mailto:serena.feng.711+...@gmail.com]
发送时间: 2018年1月22日 18:08
收件人: chenjiankun
抄送: David McBride; Serena Feng; TECH-DISCUSS OPNFV
主题: Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

Hi David & Jack,

Thanks for mentioning the task, an interface for collecting deployment results 
has been ready in TestAPI[1],
the Macro for pushing deployment result is also ready by Jelien[2], I believe 
it will facilitate all the installers' work,
currently, Julien is working on pushing daisy results to TestAPI leveraging 
that Macro, I think it will be finished soon.

And regarding how to show all the information in a table, I suggest we can take 
a look at Jack's proposal first.

@Jack, a few comments:
1. what's the opinion of jenkins id?
2. for a scenario-installer combination, some will not run once in a 
day(trigger multiple times or run in multiple pods),
in this case, a simple pass/fail will be too vague, and to facilitate the 
support of data iteration, I would suggest
leveraging 8/9/10(8 passed, 9 triggered, 10total), delete the final 
statistic line
3. how about adding a healthcheck column(functest-healthcheck test cases), to 
see if the installer meets the milestone 3.0


[1]: https://gerrit.opnfv.org/gerrit/#/c/49895/
[2]: https://gerrit.opnfv.org/gerrit/#/c/48515/

BRs
Serena

On Mon, Jan 22, 2018 at 5:02 PM chenjiankun 
<chenjiank...@huawei.com<mailto:chenjiank...@huawei.com>> wrote:
Thanks, David.

According your descriptions, I have created a demo table as below(wish I do not 
misunderstanding your meaning):

scenario

date

Jenkins

Version

Installer

Deployment

Functest

yardstick

os-nosdn-nofeature-ha

2018-01-21

Jenkins id

euphrates

compass

pass

pass

pass


2018-01-21

Jenkins id

euphrates

compass

fail

not trigger

not trigger


statistic

8/9/10
(pass:8,triggered:9, total:10)

6/7/8

6/7/8



This last line in table body is the statistics information, and lines above are 
the detailed information(and it can be folded).
The score 8/9/10 is pass/triggered/total. Total means should run, triggered 
means actually run.
Also we can add three filters:
 [cid:image003.png@01D393A2.AC65E3E0]
If you select installer as compass, then will show all data related to compass.
Iterations means last x data points to be displayed.

Does this table satisfied your requirements?

BRs,
Jack Chan
发件人: David McBride 
[mailto:dmcbr...@linuxfoundation.org<mailto:dmcbr...@linuxfoundation.org>]
发送时间: 2018年1月20日 3:07
收件人: chenjiankun
抄送: TECH-DISCUSS OPNFV; tro...@redhat.com<mailto:tro...@redhat.com>; Brattain, 
Ross B; Rao, Sridhar; OLLIVIER Cédric IMT/OLN; 
mark.bei...@dell.com<mailto:mark.bei...@dell.com>; Yuyang (Gabriel); ALFRED C 
'MORTON ' (acmor...@att.com<mailto:acmor...@att.com>); 
emma.l.fo...@intel.com<mailto:emma.l.fo...@intel.com>; Liyin (Ace); Wangwulin 
(Linda); georg.k...@ericsson.com<mailto:georg.k...@ericsson.com>; Serena Feng; 
Julien
主题: Re: [opnfv-tech-discuss][test-wg]Requirements for test resources collection

+Serena, Julien

Thanks, Jack.

  1.  Data reported per scenario (i.e., jenkins job, deployment, functest, 
yardstick, etc. displayed together for each scenario) instead of separate test 
silos.
  2.  Include deployment results
  3.  Include all Jenkins job results (failure to start, failure to complete, 
etc.)
  4.  Clear date/time stamps for every data point
  5.  Display the data above for the last x data points (e.g., 4, 5, 10 ?)
  6.  Use an easy-to-understand, unified scoring method for all test frameworks.
As I mentioned, yesterday, Julien and Serena have been working on this, as 
well.  Julien has developed a macro<https://gerrit.opnfv.org/gerrit/#/c/48515/> 
to enable consolidation of all results per scenario. He is intending to use the 
Daisy installer as a platform to verify the macro, which then can be adapted to 
other installers.

In addition, Serena has agreed to help manage an intern who can assist with the 
project.  I have an action to create an intern proposal for that purpose.

David

On Fri, Jan 19, 2018 at 1:23 AM, chenjiankun 
<chenjiank...@huawei.com<mailto

Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

[SerenaFeng(zte)]

Hi David & Jack,

Thanks for mentioning the task, an interface for collecting deployment
results has been ready in TestAPI[1],
the Macro for pushing deployment result is also ready by Jelien[2], I
believe it will facilitate all the installers' work,
currently, Julien is working on pushing daisy results to TestAPI leveraging
that Macro, I think it will be finished soon.

And regarding how to show all the information in a table, I suggest we can
take a look at Jack's proposal first.

@Jack, a few comments:
1. what's the opinion of jenkins id?
2. for a scenario-installer combination, some will not run once in a
day(trigger multiple times or run in multiple pods),
in this case, a simple pass/fail will be too vague, and to facilitate
the support of data iteration, I would suggest
leveraging 8/9/10(8 passed, 9 triggered, 10total), delete the
final statistic line
3. how about adding a healthcheck column(functest-healthcheck test cases),
to see if the installer meets the milestone 3.0


[1]: https://gerrit.opnfv.org/gerrit/#/c/49895/
[2]: https://gerrit.opnfv.org/gerrit/#/c/48515/

BRs
Serena

On Mon, Jan 22, 2018 at 5:02 PM chenjiankun <chenjiank...@huawei.com> wrote:

> Thanks, David.
>
>
>
> According your descriptions, I have created a demo table as below(wish I
> do not misunderstanding your meaning):
>
>
>
> *scenario *
>
> *date *
>
> *Jenkins *
>
> *Version *
>
> *Installer *
>
> *Deployment *
>
> *Functest *
>
> *yardstick *
>
> os-nosdn-nofeature-ha
>
> 2018-01-21
>
> Jenkins id
>
> euphrates
>
> compass
>
> pass
>
> pass
>
> pass
>
> 2018-01-21
>
> Jenkins id
>
> euphrates
>
> compass
>
> fail
>
> not trigger
>
> not trigger
>
> statistic
>
> 8/9/10
>
> (pass:8,triggered:9, total:10)
>
> 6/7/8
>
> 6/7/8
>
>
>
>
>
> This last line in table body is the statistics information, and lines
> above are the detailed information(and it can be folded).
>
> The score 8/9/10 is pass/triggered/total. Total means should run,
> triggered means actually run.
>
> Also we can add three filters:
>
>
>
> If you select installer as compass, then will show all data related to
> compass.
>
> Iterations means last x data points to be displayed.
>
>
>
> Does this table satisfied your requirements?
>
>
>
> BRs,
>
> Jack Chan
>
> *发件人:* David McBride [mailto:dmcbr...@linuxfoundation.org]
> *发送时间:* 2018年1月20日 3:07
> *收件人:* chenjiankun
> *抄送:* TECH-DISCUSS OPNFV; tro...@redhat.com; Brattain, Ross B; Rao,
> Sridhar; OLLIVIER Cédric IMT/OLN; mark.bei...@dell.com; Yuyang (Gabriel);
> ALFRED C 'MORTON ' (acmor...@att.com); emma.l.fo...@intel.com; Liyin
> (Ace); Wangwulin (Linda); georg.k...@ericsson.com; Serena Feng; Julien
> *主题:* Re: [opnfv-tech-discuss][test-wg]Requirements for test resources
> collection
>
>
>
> +Serena, Julien
>
>
>
> Thanks, Jack.
>
>1. Data reported per scenario (i.e., jenkins job, deployment,
>functest, yardstick, etc. displayed together for each scenario) instead of
>separate test silos.
>2. Include deployment results
>3. Include all Jenkins job results (failure to start, failure to
>complete, etc.)
>4. Clear date/time stamps for every data point
>5. Display the data above for the last x data points (e.g., 4, 5, 10 ?)
>6. Use an easy-to-understand, unified scoring method for all test
>frameworks.
>
> As I mentioned, yesterday, Julien and Serena have been working on this, as
> well.  Julien has developed a macro
> <https://gerrit.opnfv.org/gerrit/#/c/48515/> to enable consolidation of
> all results per scenario. He is intending to use the Daisy installer as a
> platform to verify the macro, which then can be adapted to other installers.
>
>
>
> In addition, Serena has agreed to help manage an intern who can assist
> with the project.  I have an action to create an intern proposal for that
> purpose.
>
>
>
> David
>
>
>
> On Fri, Jan 19, 2018 at 1:23 AM, chenjiankun <chenjiank...@huawei.com>
> wrote:
>
> Hi,
>
>
>
> As we discussed last test working group weekly meeting, we want to do test
> resources aggregation.
>
> We plan to offer a new friendly web portal which contain all existing test
> resource and more functions.
>
>
>
> I have a broad classification as bellow:
>
> 1.   Data analysis
>
> a) Reporting(existing, For release)
>
> b) Bitergia(existing)
>
> c) Grafana(existing, For detailed test results)
>
> d) ……(maybe we can develop more tools to show our detailed test
> results)
>
> 2.   Tes

Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

[chenjiankun]

Thanks, David.

According your descriptions, I have created a demo table as below(wish I do not 
misunderstanding your meaning):

scenario

date

Jenkins

Version

Installer

Deployment

Functest

yardstick

os-nosdn-nofeature-ha

2018-01-21

Jenkins id

euphrates

compass

pass

pass

pass


2018-01-21

Jenkins id

euphrates

compass

fail

not trigger

not trigger


statistic

8/9/10
(pass:8,triggered:9, total:10)

6/7/8

6/7/8



This last line in table body is the statistics information, and lines above are 
the detailed information(and it can be folded).
The score 8/9/10 is pass/triggered/total. Total means should run, triggered 
means actually run.
Also we can add three filters:
 [cid:image003.png@01D393A2.AC65E3E0]
If you select installer as compass, then will show all data related to compass.
Iterations means last x data points to be displayed.

Does this table satisfied your requirements?

BRs,
Jack Chan
发件人: David McBride [mailto:dmcbr...@linuxfoundation.org]
发送时间: 2018年1月20日 3:07
收件人: chenjiankun
抄送: TECH-DISCUSS OPNFV; tro...@redhat.com; Brattain, Ross B; Rao, Sridhar; 
OLLIVIER Cédric IMT/OLN; mark.bei...@dell.com; Yuyang (Gabriel); ALFRED C 
'MORTON ' (acmor...@att.com); emma.l.fo...@intel.com; Liyin (Ace); Wangwulin 
(Linda); georg.k...@ericsson.com; Serena Feng; Julien
主题: Re: [opnfv-tech-discuss][test-wg]Requirements for test resources collection

+Serena, Julien

Thanks, Jack.

  1.  Data reported per scenario (i.e., jenkins job, deployment, functest, 
yardstick, etc. displayed together for each scenario) instead of separate test 
silos.
  2.  Include deployment results
  3.  Include all Jenkins job results (failure to start, failure to complete, 
etc.)
  4.  Clear date/time stamps for every data point
  5.  Display the data above for the last x data points (e.g., 4, 5, 10 ?)
  6.  Use an easy-to-understand, unified scoring method for all test frameworks.
As I mentioned, yesterday, Julien and Serena have been working on this, as 
well.  Julien has developed a macro<https://gerrit.opnfv.org/gerrit/#/c/48515/> 
to enable consolidation of all results per scenario. He is intending to use the 
Daisy installer as a platform to verify the macro, which then can be adapted to 
other installers.

In addition, Serena has agreed to help manage an intern who can assist with the 
project.  I have an action to create an intern proposal for that purpose.

David

On Fri, Jan 19, 2018 at 1:23 AM, chenjiankun 
<chenjiank...@huawei.com<mailto:chenjiank...@huawei.com>> wrote:
Hi,

As we discussed last test working group weekly meeting, we want to do test 
resources aggregation.
We plan to offer a new friendly web portal which contain all existing test 
resource and more functions.

I have a broad classification as bellow:

1.   Data analysis

a) Reporting(existing, For release)

b) Bitergia(existing)

c) Grafana(existing, For detailed test results)

d) ……(maybe we can develop more tools to show our detailed test results)

2.   Test working group information(What information you want to see from 
test working group? Test working group event? Event of each project?)

3.   Tools of each project(Need each project member to complete)

4.   ……(waiting for you to improve)


This email is aim at collecting requirements for test resources, so if you have 
any idea about classification, existing tools(such as reporting), new functions 
you want, please do not hesitate to comment here.
As Gabriel said, he will create a new wiki page for test resources collection, 
so you can also comment there.

@David, @Tim, can you repeat your advice about reporting here? I will try my 
best to implement it.
@All, all requirements, advice, comment are welcome~☺

BRs,
Jack Chan



--
David McBride
Release Manager, OPNFV
Mobile: +1.805.276.8018<tel:%2B1.805.276.8018>
Email/Google Talk: 
dmcbr...@linuxfoundation.org<mailto:dmcbr...@linuxfoundation.org>
Skype: davidjmcbride1
IRC: dmcbride
___
opnfv-tech-discuss mailing list
opnfv-tech-discuss@lists.opnfv.org
https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss

Re: [opnfv-tech-discuss] [test-wg]Requirements for test resources collection

[David McBride]

+Serena, Julien

Thanks, Jack.

   1. Data reported per scenario (i.e., jenkins job, deployment, functest,
   yardstick, etc. displayed together for each scenario) instead of separate
   test silos.
   2. Include deployment results
   3. Include all Jenkins job results (failure to start, failure to
   complete, etc.)
   4. Clear date/time stamps for every data point
   5. Display the data above for the last x data points (e.g., 4, 5, 10 ?)
   6. Use an easy-to-understand, unified scoring method for all test
   frameworks.

As I mentioned, yesterday, Julien and Serena have been working on this, as
well.  Julien has developed a macro
 to enable consolidation of all
results per scenario. He is intending to use the Daisy installer as a
platform to verify the macro, which then can be adapted to other installers.

In addition, Serena has agreed to help manage an intern who can assist with
the project.  I have an action to create an intern proposal for that
purpose.

David

On Fri, Jan 19, 2018 at 1:23 AM, chenjiankun 
wrote:

> Hi,
>
>
>
> As we discussed last test working group weekly meeting, we want to do test
> resources aggregation.
>
> We plan to offer a new friendly web portal which contain all existing test
> resource and more functions.
>
>
>
> I have a broad classification as bellow:
>
> 1.   Data analysis
>
> a) Reporting(existing, For release)
>
> b) Bitergia(existing)
>
> c) Grafana(existing, For detailed test results)
>
> d) ……(maybe we can develop more tools to show our detailed test
> results)
>
> 2.   Test working group information(What information you want to see
> from test working group? Test working group event? Event of each project?)
>
> 3.   Tools of each project(Need each project member to complete)
>
> 4.   ……(waiting for you to improve)
>
>
>
>
>
> This email is aim at collecting requirements for test resources, so if you
> have any idea about classification, existing tools(such as reporting), new
> functions you want, please do not hesitate to comment here.
>
> As Gabriel said, he will create a new wiki page for test resources
> collection, so you can also comment there.
>
>
>
> @David, @Tim, can you repeat your advice about reporting here? I will try
> my best to implement it.
>
> @All, all requirements, advice, comment are welcome~J
>
>
>
> BRs,
>
> Jack Chan
>



-- 
*David McBride*
Release Manager, OPNFV
Mobile: +1.805.276.8018
Email/Google Talk: dmcbr...@linuxfoundation.org
Skype: davidjmcbride1
IRC: dmcbride
___
opnfv-tech-discuss mailing list
opnfv-tech-discuss@lists.opnfv.org
https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss