[JIRA] (JENKINS-21605) Logging all UpstreamCause's floods Jenkins in large setups
Title: Message Title Dirk Thomas commented on JENKINS-21605 Re: Logging all UpstreamCause's floods Jenkins in large setups Another case with the console output containing 338,653 lines with nothing more than `Started by upstream project` and `originally caused by` lines: http://build.ros.org/job/Mrel_sync-packages-to-testing_bionic_amd64/529/ Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-21605) Logging all UpstreamCause's floods Jenkins in large setups
Title: Message Title Dirk Thomas updated an issue Jenkins / JENKINS-21605 Logging all UpstreamCause's floods Jenkins in large setups Change By: Dirk Thomas URL: http:// jenkins build .ros.org/job/ ros Mrel_sync - hydro packages - metapackages_binarydeb_precise_amd64 to-testing_bionic_amd64 / 813 529 / Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38215) msbuild compiler warning not detected / extracted
Title: Message Title Dirk Thomas commented on JENKINS-38215 Re: msbuild compiler warning not detected / extracted Thank you very much for addressing this. I am looking forward for the next release to use it. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38215) msbuild compiler warning not detected / extracted
Title: Message Title Dirk Thomas commented on JENKINS-38215 Re: msbuild compiler warning not detected / extracted Yes, that is the only warning not being recognized (the same warning appears twice in the console log). In contrast to the cases in your test which are all file specific these ones are warning for the command line arguments. Therefore it doesn't start with a filename / row / column but just with `Command line warning ...`. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38215) msbuild compiler warning not detected / extracted
Title: Message Title Dirk Thomas created an issue Jenkins / JENKINS-38215 msbuild compiler warning not detected / extracted Issue Type: Bug Assignee: Ulli Hafner Components: warnings-plugin Created: 2016/Sep/14 8:23 PM Environment: Jenkins 2.7.1, Warnings Plug-in 4.56, Master is an Ubuntu 14.04 system, Slave is a Windows 10 system Priority: Minor Reporter: Dirk Thomas The console out of the job http://ci.ros2.org/job/ci_windows/1651/consoleFull shows the following compiler warning: (ClCompile target) -> cl : Command line warning D9002: ignoring unknown option '-std=c++11' but the result says: "MSBuild Warnings: 0 warnings". I am happy to provide more information if necessary and/or try a from-source build to check if it makes any difference. Add Comment
[JIRA] (JENKINS-32148) highlight test result lines on hover
Title: Message Title Dirk Thomas commented on JENKINS-32148 Re: highlight test result lines on hover Please see https://github.com/jenkinsci/junit-plugin/pull/55 for a patch implementing this. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-15748) fix Executor.interrupt(ABORTED) to let Groovy script cancel a build
Title: Message Title Dirk Thomas commented on JENKINS-15748 Re: fix Executor.interrupt(ABORTED) to let Groovy script cancel a build This ticket is about marking a build as "aborted". Your workaround marks the build as "failed" which is not the the goal of this ticket. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-5150) Race condition for feature "Block build when upstream project is building"
Title: Message Title Dirk Thomas commented on JENKINS-5150 Re: Race condition for feature "Block build when upstream project is building" I think we have the same problem with the current LTS 1.642.1. Please see the detailed description of our case: https://github.com/ros-infrastructure/ros_buildfarm/pull/194 Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [jobconfighistory-plugin] (JENKINS-21600) Enable paging to improve performance
Title: Message Title Dirk Thomas commented on JENKINS-21600 Re: Enable paging to improve performance For a specific job the history seems to be limited based on the config value. But the global page (e.g. "Show job configs only") is not restricted at all. It would be great if pagination (or at least a configurable limit) could be added for that. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-16871) Inconsistent use of XML escaping in config.xml when uploading through API
Title: Message Title Dirk Thomas commented on JENKINS-16871 Re: Inconsistent use of XML escaping in config.xml when uploading through API When using tool like the Jenkins plugin JobConfigHistory to look at the diffs the workaround only generates more noise. It would be really great if this could be addressed at some point. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-16871) Inconsistent use of XML escaping in config.xml when uploading through API
Title: Message Title Dirk Thomas edited a comment on JENKINS-16871 Re: Inconsistent use of XML escaping in config.xml when uploading through API When using tool like the Jenkins plugin JobConfigHistory to look at the diffs the workaround only generates more noise. It would be really great if this could be addressed at some point. Btw. the behavior is still the same in the current LTS release 1.642.1. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-21605) Logging all UpstreamCause's floods Jenkins in large setups
Title: Message Title Dirk Thomas commented on JENKINS-21605 Re: Logging all UpstreamCause's floods Jenkins in large setups Since the previous URL is about to become unavailable I will add a current one which demonstrates the ridiculous amount of redundant information being logged: http://build.ros.org/job/Ibin_uT64__desktop_full__ubuntu_trusty_amd64__binary/1/ Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [junit-plugin] (JENKINS-32148) highlight lines on hover
Title: Message Title
Dirk Thomas created an issue
Jenkins / JENKINS-32148
highlight lines on hover
Issue Type:
Improvement
Assignee:
Unassigned
Components:
junit-plugin
Created:
20/Dec/15 12:50 AM
Priority:
Trivial
Reporter:
Dirk Thomas
The tables ("All failed tests" and "All Tests") can be very wide. When looking at the test names in the first column it is not that easy to see which values in the other columns belong to the same row. It would be a great improvement if each line would highlight (change its background color) on hover.
Something similar already happens for each build in the build history on the left pane and in the parameter form when triggering a parameterized build.
Add Comment
[JIRA] [junit-plugin] (JENKINS-32148) highlight test result lines on hover
Title: Message Title Dirk Thomas updated an issue Jenkins / JENKINS-32148 highlight test result lines on hover Change By: Dirk Thomas Summary: highlight test result lines on hover Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [xunit-plugin] (JENKINS-31699) use testsuite name to built hierachy
Title: Message Title
Dirk Thomas updated an issue
Jenkins / JENKINS-31699
use testsuite name to built hierachy
Change By:
Dirk Thomas
I am using `nose` to generate xunit test result files. With the option `--xunit-testsuite-name=` (https://nose.readthedocs.org/en/latest/plugins/xunit.html#cmdoption--xunit-testsuite-name) it sets the `name` attribute in the `testsuite` tag to a custom value instead of the default value `nosetests`). The following is an example result file - note the `name` attribute in the root tag:{code:xml} {code}Currently the xunit plugin (I am using the JUnit repor) does not use the `name` attribute of the `testsuite` tag anywhere. The hierarchy of the test results is built only based on the `classname` of the `testcase` tag. If multiple result files contain testcases it is not clear to which result file they belong.It would be very helpful if the hierarchy would also include the `name` attribute of the `testsuite` tag to build the hierarchy. This would group the testcases from each testsuite more reasonable.Please let me know if I can provide any more information.
Add Comment
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this
[JIRA] [junit-plugin] (JENKINS-31495) Jobs fail when no XML files found
Title: Message Title Dirk Thomas commented on JENKINS-31495 Re: Jobs fail when no XML files found This has been requested before: https://issues.jenkins-ci.org/browse/JENKINS-12815 I guess you have the same problem that your generated job can't know if the build will produce test result files or not. Sadly until now this is not possible. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [xunit-plugin] (JENKINS-31699) use testsuite name to built hierachy
Title: Message Title Dirk Thomas created an issue Jenkins / JENKINS-31699 use testsuite name to built hierachy Issue Type: Improvement Assignee: Gregory Boissinot Components: xunit-plugin Created: 21/Nov/15 10:10 PM Environment: Jenkins 1.638 xUnit plugin 1.98 Ubuntu 14.04 nose 1.3.7 Labels: xunit junit nose Priority: Minor Reporter: Dirk Thomas I am using `nose` to generate xunit test result files. With the option `--xunit-testsuite-name=` (https://nose.readthedocs.org/en/latest/plugins/xunit.html#cmdoption--xunit-testsuite-name) it sets the `name` attribute in the `testsuite` tag to a custom value instead of the default value `nosetests`). The following is an example result file - note the `name` attribute in the root tag: "ament_package.nosetests" tests="2" errors="0" failures="0" skip="0"> "test.test_something.FooTest" name="test_foo" time="0.001"> "test.test_something.BarTest" name="test_bar"
[JIRA] [junit-plugin] (JENKINS-31699) use testsuite name to built hierachy
Title: Message Title Dirk Thomas updated an issue Jenkins / JENKINS-31699 use testsuite name to built hierachy Change By: Dirk Thomas Component/s: junit-plugin Environment: Jenkins 1.638 JUnit plugin 1.9 xUnit plugin 1.98 Ubuntu 14.04 nose 1.3.7 Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [collapsing-console-sections-plugin] (JENKINS-25201) list of console section not scrollable, in case of many section the later ones are unaccessible
Title: Message Title Dirk Thomas commented on JENKINS-25201 Re: list of console section not scrollable, in case of many section the later ones are unaccessible I have created a pull request to fix this problem: https://github.com/jenkinsci/collapsing-console-sections-plugin/pull/6 Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [collapsing-console-sections-plugin] (JENKINS-25201) list of console section not scrollable, in case of many section the later ones are unaccessible
Title: Message Title Dirk Thomas commented on JENKINS-25201 Re: list of console section not scrollable, in case of many section the later ones are unaccessible I have created a pull request to fix this problem: https://github.com/jenkinsci/collapsing-console-sections-plugin/pull/6 Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-15283) The node does not return back online as there is enough disk space again
Dirk Thomas commented on JENKINS-15283 The node does not return back online as there is enough disk space again I recently ran into the same problem: Several slaves were marked "offline temporarily due to the lack of disk space". While in the meantime the disk space has been recovered they were not reenabled automatically for several days. After clicking the "refresh" button in the nodes view they reenabled fine. I am currently using Jenkins 1.594. Which plugin are you referring to which fixed the problem between version 2.41 and 2.43. I couldn't find any plugin on the system with a similar version range. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [dashboard-view-plugin] (JENKINS-26961) build statistics which only consider the latest build of each job
Dirk Thomas created JENKINS-26961 build statistics which only consider the latest build of each job Issue Type: New Feature Assignee: Peter Hayes Components: dashboard-view-plugin Created: 13/Feb/15 9:25 PM Description: Currently the build statistics aggregate the numbers from the last ten builds of each job. But I would like to see only the "current" stats - it doesn't matter to be if a build had a different state before. If the max number of builds could be configured to e.g. "1" that would enable this use case. Project: Jenkins Priority: Minor Reporter: Dirk Thomas This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [junit-plugin] (JENKINS-12815) Add an option to select Build status when no Test is found
Dirk Thomas commented on JENKINS-12815 Add an option to select Build status when no Test is found +1 since I generate jobs from a template from thousands of projects I simply don't know if a project will generate test results files or not. Using the `xunit` plugin instead (which has the requested configuration option) does not work for me either since it is much stricter when validating the xml files. As a temporary fix I conditionally generate a dummy result file. But that "pollutes" all test statistics. I do understand the fine line of flexibility vs. bloated but this seems to break a use case several users are having. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-21605) Logging all UpstreamCause's floods Jenkins in large setups
Dirk Thomas
edited a comment on JENKINS-21605
Logging all UpstreamCauses floods Jenkins in large setups
I will describe two concrete cases to have a baseline for the further discussion.
Case (A) ( https://gist.github.com/dirk-thomas/9bbd47397e48ef3ceef8 ):
A job "leaf" has only a single upstream dependency on "before_leaf".
And "before leaf" has many (in this example 40: "01" to "40") upstream dependencies.
Each upstream dependency "N" has "N-1" as its upstream dependency.
The "before_leaf" job will list all 40 upstream causes.
Each upstream cause on its own is limited to a recursive depth of 10 (according to `MAX_DEPTH`).
The "leaf" job has a single upstream cause ("before_leaf").
The `SetString traversed` in the `UpstreamClause` prevents listing repeated upstream causes of the single upstream cause.
Case (B) ( https://gist.github.com/dirk-thomas/37febb42abeb8631f946 ):
A job "leaf" has only a single upstream dependency on "before_leaf".
And "before leaf" has several (in this example 5: "a15" to "e15") upstream dependencies.
Each upstream dependency "xN" has "xN-1" as its upstream dependency.
Recursive upstream causes are usually "terminated" by a `DeeplyNestedUpstreamCause` when `MAX_DEPTH` is reached.
`MAX_LEAF` prevents adding a `DeeplyNestedUpstreamCause` at the end of the recursion once the number of different causes has reached 25 addresses (`MAX_LEAF`).
This can be seen in the "leaf" of of case (B).
(I don't understand why skipping the `DeeplyNestedUpstreamCause` when aborting the recursion makes a big different though - it does not affect the log size significantly and contains valuable information (that the recursion has been aborted)).
Based on these I identified two problems.
Problem (A): limitation of performing the thresholds in the `UpstreamCause`:
The "before_leaf" job of case (A) has 40 upstream causes.
While each on its own does some logic for limiting the information each separate `UpstreamCause` instance does not know about its siblings.
Therefore it can not adjust the level of information shown in the case that there are many siblings.
This is not "fixable" in the `UpstreamCause` class itself.
This would require some changes in the code handling the upstream causes to pass in information e.g. the number of siblings (which arguably a `UpstreamCause` should not need to know about).
(The problem is the same for the "before_leaf" job of case (B).)
Problem (B): the depth threshold is independent from the number of upstream causes:
The "leaf" job of case (B) has only a single upstream cause.
But this upstream cause outputs every upstream cause up to the recursion limit.
This results in N x 10 upstream causes where N is the number of upstream causes of the single upstream cause of the job.
A "combined" limit would probably make much more sense in this case.
E.g. limit each recursion to not 10 but potentially less if the number of sibling upstream causes on the first level increases.
(I am unable to provide a Java unit test since I lack the experience programming in Java but the Groovy examples should be verify specific and hopefully easy to transfer into a unit test by an experienced Jenkins/Java programmer.)
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
--
You received this message because you are subscribed to the Google Groups Jenkins Issues group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-21605) Logging all UpstreamCause's floods Jenkins in large setups
Dirk Thomas commented on JENKINS-21605 Logging all UpstreamCauses floods Jenkins in large setups Can someone provide some insight how the problem (A) could be addressed? If we could sketch the desired solution together I might try to work on a patch for it. But currently the algorithmic approach is not clear to me - especially how to perform the limiting without affecting to many external classes (outside of `UpstreamCause`. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-21605) Logging all UpstreamCause's floods Jenkins in large setups
Dirk Thomas
edited a comment on JENKINS-21605
Logging all UpstreamCauses floods Jenkins in large setups
I will describe two concrete cases to have a baseline for the further discussion.
Case (A) ( https://gist.github.com/dirk-thomas/9bbd47397e48ef3ceef8 ):
A job "leaf" has only a single upstream dependency on "before_leaf".
And "before leaf" has many (in this example 40: "01" to "40") upstream dependencies.
Each upstream dependency "N" has "N-1" as its upstream dependency.
The "before_leaf" job will list all 40 upstream causes.
Each upstream cause on its own is limited to a recursive depth of 10 (according to `MAX_DEPTH`).
The "leaf" job has a single upstream cause ("before_leaf").
The `SetString traversed` in the `UpstreamClause` prevents listing repeated upstream causes of the single upstream cause.
Case (B) ( https://gist.github.com/dirk-thomas/37febb42abeb8631f946 ):
A job "leaf" has only a single upstream dependency on "before_leaf".
And "before leaf" has several (in this example 5: "a15" to "e15") upstream dependencies.
Each upstream dependency "xN" has "xN-1" as its upstream dependency.
Recursive upstream causes are usually "terminated" by a `DeeplyNestedUpstreamCause` when `MAX_DEPTH` is reached.
`MAX_LEAF` prevents adding a `DeeplyNestedUpstreamCause` at the end of the recursion once the number of different causes has reached 25 addresses (`MAX_LEAF`).
This can be seen in the "leaf" of of case (B).
(I don't understand why skipping the `DeeplyNestedUpstreamCause` when aborting the recursion makes a big different though - it does not affect the log size significantly and contains valuable information (that the recursion has been aborted)).
Based on these I identified two problems.
Problem (A): limitation of performing the thresholds in the `UpstreamCause`:
The "before_leaf" job of case (A) has 40 upstream causes.
While each on its own does some logic for limiting the information each separate `UpstreamCause` instance does not know about its siblings.
Therefore it can not adjust the level of information shown in the case that there are many siblings.
This is not "fixable" in the `UpstreamCause` class itself.
This would require some changes in the code handling the upstream causes to pass in information e.g. the number of siblings (which arguably a `UpstreamCause` should not need to know about).
(The problem is the same for the "before_leaf" job of case (B).)
Problem (B): the depth threshold is independent from the number of upstream causes:
The "leaf" job of case (B) has only a single upstream cause.
But this upstream cause outputs every upstream cause up to the recursion limit.
This results in N x 10 upstream causes where N is the number of upstream causes of the single upstream cause of the job.
I "combined" limit would probably make much more sense in this case.
E.g. limit each recursion to not 10 but potentially less if the number of sibling upstream causes on the first level increases.
(I am unable to provide a Java unit test since I lack the experience programming in Java but the Groovy examples should be verify specific and hopefully easy to transfer into a unit test by an experienced Jenkins/Java programmer.)
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
--
You received this message because you are subscribed to the Google Groups Jenkins Issues group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-21605) Logging all UpstreamCause's floods Jenkins in large setups
Dirk Thomas
commented on JENKINS-21605
Logging all UpstreamCauses floods Jenkins in large setups
I will describe two concrete cases to have a baseline for the further discussion.
Case (A) (https://gist.github.com/dirk-thomas/9bbd47397e48ef3ceef8):
A job "leaf" has only a single upstream dependency on "before_leaf".
And "before leaf" has many (in this example 40: "01" to "40") upstream dependencies.
Each upstream dependency "N" has "N-1" as its upstream dependency.
The "before_leaf" job will list all 40 upstream causes.
Each upstream cause on its own is limited to a recursive depth of 10 (according to `MAX_DEPTH`).
The "leaf" job has a single upstream cause ("before_leaf").
The `SetString traversed` in the `UpstreamClause` prevents listing repeated upstream causes of the single upstream cause.
Case (B) (https://gist.github.com/dirk-thomas/37febb42abeb8631f946):
A job "leaf" has only a single upstream dependency on "before_leaf".
And "before leaf" has several (in this example 5: "a15" to "e15") upstream dependencies.
Each upstream dependency "xN" has "xN-1" as its upstream dependency.
Recursive upstream causes are usually "terminated" by a `DeeplyNestedUpstreamCause` when `MAX_DEPTH` is reached.
`MAX_LEAF` prevents adding a `DeeplyNestedUpstreamCause` at the end of the recursion once the number of different causes has reached 25 addresses (`MAX_LEAF`).
This can be seen in the "leaf" of of case (B).
(I don't understand why skipping the `DeeplyNestedUpstreamCause` when aborting the recursion makes a big different though - it does not affect the log size significantly and contains valuable information (that the recursion has been aborted)).
Based on these I identified two problems.
Problem (A): limitation of performing the thresholds in the `UpstreamCause`:
The "before_leaf" job of case (A) has 40 upstream causes.
While each on its own does some logic for limiting the information each separate `UpstreamCause` instance does not know about its siblings.
Therefore it can not adjust the level of information shown in the case that there are many siblings.
This is not "fixable" in the `UpstreamCause` class itself.
This would require some changes in the code handling the upstream causes to pass in information e.g. the number of siblings (which arguably a `UpstreamCause` should not need to know about).
(The problem is the same for the "before_leaf" job of case (B).)
Problem (B): the depth threshold is independent from the number of upstream causes:
The "leaf" job of case (B) has only a single upstream cause.
But this upstream cause outputs every upstream cause up to the recursion limit.
This results in N x 10 upstream causes where N is the number of upstream causes of the single upstream cause of the job.
I "combined" limit would probably make much more sense in this case.
E.g. limit each recursion to not 10 but potentially less if the number of sibling upstream causes on the first level increases.
(I am unable to provide a Java unit test since I lack the experience programming in Java but the Groovy examples should be verify specific and hopefully easy to transfer into a unit test by an experienced Jenkins/Java programmer.)
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
--
You received this message because you are subscribed to the Google Groups Jenkins Issues group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-21605) Logging all UpstreamCause's floods Jenkins in large setups
Dirk Thomas commented on JENKINS-21605 Logging all UpstreamCauses floods Jenkins in large setups I have created a Groovy script which creates some jobs which demonstrate the unbounded breadth you mentioned: https://gist.github.com/dirk-thomas/630084eefb44baa79f15 An example job can also be found here: http://54.183.26.131:8080/view/jenkins21605/job/jenkins21605_leaf/1/ This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-21605) Logging all UpstreamCause's floods Jenkins in large setups
Dirk Thomas edited a comment on JENKINS-21605 Logging all UpstreamCauses floods Jenkins in large setups I have updated the Groovy script to generate a job topology like this: 01 depends on root 02 depends on 01 ... 40 depends on 39 leaf depends on: 01-40 The result can be found here: http://54.183.26.131:8080/view/jenkins21605/job/jenkins21605_leaf/lastSuccessfulBuild/ The build log is already 84 KB in size (just for logging the upstream causes). I think the "best" approach to address this is not to add another threshold but simply give the user the option not to log the recursive upstream cause. It is perfectly fine to list all "Started by upstream project" lines for all the upstream dependencies. But optionally not outputting the ever increasing "originally caused by" would reduce the output to linear size (linear to the number of upstream dependencies). And the user can still navigate the hierarchy of upstream causes by following the upstream build links. Do you agree that this test is sufficient? Do you agree that the proposed option would be a reasonable approach to address the problem? This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-21605) Logging all UpstreamCause's floods Jenkins in large setups
Dirk Thomas edited a comment on JENKINS-21605 Logging all UpstreamCauses floods Jenkins in large setups I have updated the Groovy script to generate a job topology like this: 01 depends on root 02 depends on 01 ... 40 depends on 39 leaf depends on: 01-40 The result can be found here: http://54.183.26.131:8080/view/jenkins21605/job/jenkins21605_leaf/lastSuccessfulBuild/ The build log is already 84 KB in size (just for logging the upstream causes). I think the "best" approach to address this is not to add another threshold but simply give the user the option not to log the recursive upstream cause. It is perfectly fine to list all "Started by upstream project" lines for all the upstream dependencies. But optionally not outputting the ever increasing "originally caused by" would reduce the output to linear size (linear to the number of upstream dependencies). And the user can still navigate the hierarchy of upstream causes by following the upstream build links. Do you agree that this test is sufficient? Do you agree that the proposed option would be a reasonable approach to address the problem or do you see a better approach? This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-21605) Logging all UpstreamCause's floods Jenkins in large setups
Dirk Thomas commented on JENKINS-21605 Logging all UpstreamCauses floods Jenkins in large setups But it should show ALL direct upstream projects which triggered the build. And in combination with that the current threshold for the recursion depth (of 11) is already resulting in too big output. So what kind of threshold are you proposing then? This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-21605) Logging all UpstreamCause's floods Jenkins in large setups
Dirk Thomas commented on JENKINS-21605 Logging all UpstreamCauses floods Jenkins in large setups I have updated the Groovy script to generate a job topology like this: 01 depends on root 02 depends on 01 ... 40 depends on 39 leaf depends on: 01-40 The result can be found here: http://54.183.26.131:8080/view/jenkins21605/job/jenkins21605_leaf/lastSuccessfulBuild/ The build log is already 84 KB in size (just for logging the upstream causes). I think the "best" approach to address this is not to add another threshold but simply give the user the option not to log the recursive upstream cause. It is perfectly fine to list all "Started by upstream project" lines for all the upstream dependencies. But optionally not outputting the ever increasing "originally caused by" would reduce the output to linear size (linear to the number of upstream dependencies). Do you agree that this test is sufficient? Do you agree that the proposed option would be a reasonable approach to address the problem? This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-21605) Logging all UpstreamCause's floods Jenkins in large setups
Dirk Thomas commented on JENKINS-21605 Logging all UpstreamCauses floods Jenkins in large setups Obviously the logged data is highly redundant. And this bug has been fixed before. The actual output of the job is only a few KB so exploding it to hundreds of MB is a serious misuse of resources. Our Jenkins deployments have tens of thousands of jobs which have a lot of downstream dependencies between them. The longer these dependency chains get to longer and more redundant the output gets. It does not only scale linear with the dependency hierarchy but exponentially because of the many repetitions of the same dependencies. (Why are we doing that? Each job build a certain Debian package and afterwards triggers its downstream dependencies since we can not guarantee ABI stability. ) The example job is one of the many but each and every job has the same problem. The deeper in the dependency graph the more severe the problem is. Since we trigger these jobs multiple times per day that results in a lot of GBs of unnecessary data which keep growing with every job we run and the master has to keep them. We don't want to truncate the number of kept builds because we value the information available in the (useful part of the) build log. We also can't provide unlimited fast storage to the master. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-21605) Logging all UpstreamCause's floods Jenkins in large setups
Dirk Thomas commented on JENKINS-21605 Logging all UpstreamCauses floods Jenkins in large setups The argument that the "console tail shown by default includes everything that's relevant" is simply wrong in a lot of cases. What if the content of interest is just a little bit above the tail? As you pointed out it is basically not feasible to view it in the browser anymore. But why do I have to download hundreds of MBs which basically contain 99% garbage? This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-21605) Logging all UpstreamCause's floods Jenkins in large setups
Dirk Thomas commented on JENKINS-21605 Logging all UpstreamCauses floods Jenkins in large setups If you would like to reduce the severity e.g. to "Major" please feel free to do so. But I disagree on it being only an "inconvenience" which is not worth to be addressed. For us it makes Jenkins almost unusable for large scale deployments. Either our Jenkins master dies at some point because it runs out of storage or we have to accept "loss of data" in terms of removing build logs way earlier than it should be necessary. Arguably Jenkins would benefit from a global object to not print upstream triggers at all. The same as SVN might benefit from an option to be less verbose about what it checked out. But the current listing of highly redundant upstream triggers is simply wrong. It (the redundant part) does not contain any information at all! The especially disappointing part is that it was fixed before and nobody seems to care about the regression for now a year. I fear that this will not change with the severity lowered and the ticket will remain unresolved forever. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-21605) Logging all UpstreamCause's floods Jenkins in large setups
Dirk Thomas commented on JENKINS-21605 Logging all UpstreamCauses floods Jenkins in large setups Thank you for pointing out potential workarounds. The compression might be an option but also add additional load on the master. I don't know if that would be acceptable without further testing. Replacing the project dependencies with a custom trigger mechanism wouldn't be difficult (since we generate the jobs anyway) but I don't want to remove that valuable information. This seems to be working around core Jenkins elements. Also the trigger reason in the log is nice - if it would not contain the exponentially exploded redundant information. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
