[
https://issues.apache.org/jira/browse/BEAM-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-4298:
--------------------------------
Labels: stale-P2 triaged (was: triaged)
> Perfkit runs are always "official" while running them with Jenkins
> ------------------------------------------------------------------
>
> Key: BEAM-4298
> URL: https://issues.apache.org/jira/browse/BEAM-4298
> Project: Beam
> Issue Type: Bug
> Components: testing
> Reporter: Lukasz Gajowy
> Priority: P2
> Labels: stale-P2, triaged
>
> +What is "official" flag?+
> "Official" flag is a PerfkitBenchmarker boolean flag that is set in Jenkins.
> Perfkit uses it to determine whether test results should be treated as
> official ones. Currently, in the whole Performance Testing Framework it means
> that such results will be:
> - displayed in the Performance Testing Dashboards
> ([https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688])
> - used for detecting anomalies in those dashboards and reporting them to the
> community (this is work in progress at the time of writing)
> +How can Performance tests be run right now?+
> Currently, we have two options for running performance tests:
> - run them periodically (this is done by Jenkins on master branch 4 times a
> day)
> - trigger them on demand from Pull Request - for example by typing "Run Java
> TextIO Performance Test" on Github comment in the PR. For now, they also use
> master for building the code, which is useless in terms of running them in
> the PR. They should use the branches code instead otherwise it's misguiding
> for the user and error-prone. This issue is addressed here:
> https://issues.apache.org/jira/browse/BEAM-4140
> +What is the problem?+
> We shouldn't mark on demand runs as "official", because those are run to test
> unmerged code. This will pollute the dashboards with unwanted results on the
> plots. in the future, when anomaly detection is merged, it will discover
> false-positive anomalies (based on unmerged code).
> +Proposed solution:+
> The official flag in Jenkins should be set based on "GIT_BRANCH" environment
> variable (AFAIK it's out of the box in Jenkins). Only results from master
> should be official.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
