[JIRA] (JENKINS-49406) Prototype the Evergreen snapshotting data safety system
Title: Message Title Baptiste Mathus stopped work on JENKINS-49406 Change By: Baptiste Mathus Status: In Progress Open Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49406) Prototype the Evergreen snapshotting data safety system
Title: Message Title SCM/JIRA link daemon commented on JENKINS-49406 Re: Prototype the Evergreen snapshotting data safety system Code changed in jenkins User: R. Tyler Croy Path: jep/302/README.adoc jep/README.adoc http://jenkins-ci.org/commit/jep/949cbdb6bb2823a0a780e1005cf86a9b815f48b6 Log: Merge pull request #67 from batmat/JENKINS-49406-JEP-submission JENKINS-49406 Evergreen snapshotting data safety system JEP Compare: https://github.com/jenkinsci/jep/compare/b5b57a9f1c93...949cbdb6bb28 Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49406) Prototype the Evergreen snapshotting data safety system
Title: Message Title SCM/JIRA link daemon commented on JENKINS-49406 Re: Prototype the Evergreen snapshotting data safety system Code changed in jenkins User: Baptiste Mathus Path: jep//README.adoc http://jenkins-ci.org/commit/jep/6773edbc06488de4c2fa7371f54c79df38672861 Log: JENKINS-49406 Evergreen snapshotting data safety system JEP Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49406) Prototype the Evergreen snapshotting data safety system
Title: Message Title Baptiste Mathus commented on JENKINS-49406 Re: Prototype the Evergreen snapshotting data safety system Tried to start implementing the root separation by changing the "builds" and "workspace" directories as described in https://github.com/batmat/jep/blob/a3d70917b1095ee27c292c029593f79913ff186a/jep/302/README.adoc#segregate-job-configuration-and-build-data using CasC to also test/prototype this part of the proposal, but this proved impossible. See https://github.com/jenkinsci/configuration-as-code-plugin/issues/151 Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49406) Prototype the Evergreen snapshotting data safety system
Title: Message Title Baptiste Mathus commented on JENKINS-49406 Re: Prototype the Evergreen snapshotting data safety system FTR, meeting added in the repo as we'll do for all of them in the future: https://github.com/jenkins-infra/evergreen/tree/master/docs/meetings/2018-03-18-JENKINS-49406-quality-bar Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49406) Prototype the Evergreen snapshotting data safety system
Title: Message Title Raul Arabaolaza commented on JENKINS-49406 Re: Prototype the Evergreen snapshotting data safety system I fully agree also. Just for openness and even if this has been already add to other sources this are the meeting notes of my conversation with Baptiste Mathus yesterday: RAUL: This is intended for development time, not for deployment validation Idea is Try an upgrade, test all works properly perform a rollback and test again all is working BAPTISTE: We are likely to be able to reuse the “health check” logic that will have to be developed for evergreen-client itself in production, to check if Jenkins is running fine. RAUL: critical: we need to test the health check QUESTION: Should we try to implement synthetic transactions here or go with ATH which already exists? PROPOSALS for Rollback testing: Make sure there is enough coverage that all possible rollback paths are covered Create a quality bar for rollbacks Make sure you are including some failing scenarios in the quality bar Not only test the happy path, for example: Made a failed upgrade, test that we are able to detect the upgrade as a failure, rollback and test that the instance is working perfectly Made a failed upgrade, test that we are able to detect the upgrade as a failure, made a failed rollback and test that we are able to detect the rollback failed Make sure that in case of different chained rollback strategies we test each and every one of them Create a healthcheck url to be invoked via CURL for example We can create a plugin that provides that healthcheck url and integrate with ST Maybe some work from metrics plugin can be reused Some possible testing flows: Upgrade run health check (ST), rollback, ST again ¿and ATH? No work yet on ST that I am aware of, but ST can be later reused for deployment testing Run ATH, rollback, ATH again Some work already done, but ATH is maybe too heavy and coverage is pretty poor and based on individual
[JIRA] (JENKINS-49406) Prototype the Evergreen snapshotting data safety system
Title: Message Title R. Tyler Croy commented on JENKINS-49406 Re: Prototype the Evergreen snapshotting data safety system Baptiste Mathus, regarding a JEP for determining Jenkins healthiness for Jenkins Essentials, I think that's a good idea and will be a useful design document to discuss with the broader development community. Will you file a ticket for that and drop it into Milestone 1? Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49406) Prototype the Evergreen snapshotting data safety system
Title: Message Title
Baptiste Mathus edited a comment on JENKINS-49406
Re: Prototype the Evergreen snapshotting data safety system
{quote}I wonder if starting in quiet mode would result in us missing any potential errors? If not, then I say let's do it!{quote}Definitely. And [~jglick] already had a similar comment reviewing [https://github.com/batmat/jep/pull/1] But I think it would still be interesting to triage the potential issue causes, with a slightly more progressive process.Roughly, would/could be: * set to start in quiet mode next time, and restart * check Jenkins is healthy [1] * if yes, cancel quiet [EDIT: or better, write some plugin that would *only* allow our smoke testing job, on the next bullet point, to run] * start some kind of smoke testing build * if success, then \o/, if not, roll back.[1] [~rtyler] about that, I have been starting to think since a few days we probably need a dedicated JIRA/JEP to design what "evergreen-client decides if Jenkins is healthy [enough] or not", aka to trigger a rollback, or not... Do we something like this? WDYT?
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49406) Prototype the Evergreen snapshotting data safety system
Title: Message Title Baptiste Mathus commented on JENKINS-49406 Re: Prototype the Evergreen snapshotting data safety system I wonder if starting in quiet mode would result in us missing any potential errors? If not, then I say let's do it! Definitely. And Jesse Glick already had a similar comment reviewing https://github.com/batmat/jep/pull/1 But I think it would still be interesting to triage the potential issue causes, with a slightly more progressive process. Roughly, would/could be: set to start in quiet mode next time, and restart check Jenkins is healthy [1] if yes, cancel quiet start some kind of smoke testing build if success, then \o/, if not, roll back. [1] R. Tyler Croy about that, I have been starting to think since a few days we probably need a dedicated JIRA/JEP to design what "evergreen-client decides if Jenkins is healthy [enough] or not", aka to trigger a rollback, or not... Do we something like this? WDYT? Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed
[JIRA] (JENKINS-49406) Prototype the Evergreen snapshotting data safety system
Title: Message Title R. Tyler Croy commented on JENKINS-49406 Re: Prototype the Evergreen snapshotting data safety system That's an interesting idea Baptiste Mathus! I wonder if starting in quiet mode would result in us missing any potential errors? If not, then I say let's do it! Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49406) Prototype the Evergreen snapshotting data safety system
Title: Message Title Baptiste Mathus commented on JENKINS-49406 Re: Prototype the Evergreen snapshotting data safety system We had a chat today with Raul Arabaolaza and he vented a quite important thing we might want to do IMO: to reduce the risk of creating more things than necessary when Jenkins starts again after an upgrade, puting it in quiet start mode could help. Only once the evergreen client has performed the upgrade, and checked Jenkins is judged healthy, would it automatically cancel its quiet mode. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49406) Prototype the Evergreen snapshotting data safety system
Title: Message Title Baptiste Mathus commented on JENKINS-49406 Re: Prototype the Evergreen snapshotting data safety system As discussed yesterday, first draft submitted for review to the dev list: https://groups.google.com/forum/?utm_medium=email_source=footer#!topic/jenkinsci-dev/XdXuMFLXKPw (=> https://github.com/batmat/jep/pull/1) Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49406) Prototype the Evergreen snapshotting data safety system
Title: Message Title Baptiste Mathus commented on JENKINS-49406 Re: Prototype the Evergreen snapshotting data safety system Also think carefully about compatibleSinceVersion. Jesse Glick I didn't plan anything specific to be honest using this metadata. Because yes, we probably could do some optimizations on this front, for instance not reverting to previous if compatibleSinceVersion stayed the same. But as you said too, IIUC, yesterday well this practice is not currently used often and carefully enough to be really usable automatically I suspect? But agreed this might be something we can improve over time while defining the efforts and things a given plugin has to comply with to be able to enter the set of plugins delivered/used in Essentials. WDYT? (Should we rather take this in a dedicated thread on the ML BTW? I plan one anyway, so maybe we'll get back to it there very soon.) Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49406) Prototype the Evergreen snapshotting data safety system
Title: Message Title Jesse Glick commented on JENKINS-49406 Re: Prototype the Evergreen snapshotting data safety system Also think carefully about compatibleSinceVersion. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49406) Prototype the Evergreen snapshotting data safety system
Title: Message Title Jesse Glick commented on JENKINS-49406 Re: Prototype the Evergreen snapshotting data safety system For inspiration: etckeeper Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49406) Prototype the Evergreen snapshotting data safety system
Title: Message Title Baptiste Mathus started work on JENKINS-49406 Change By: Baptiste Mathus Status: Open In Progress Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49406) Prototype the Evergreen snapshotting data safety system
Title: Message Title R. Tyler Croy updated an issue Jenkins / JENKINS-49406 Prototype the Evergreen snapshotting data safety system Change By: R. Tyler Croy Sprint: Essentials - Milestone 1 Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-49406) Prototype the Evergreen snapshotting data safety system
Title: Message Title R. Tyler Croy assigned an issue to Baptiste Mathus I'm going to assign this to Baptiste Mathus. Feel free to spin up some separate tickets as necessary to explore additional avenues of experimentation. I would expect that the end-result of the prototype/experiment phase would be a JEP document. Jenkins / JENKINS-49406 Prototype the Evergreen snapshotting data safety system Change By: R. Tyler Croy Assignee: R. Tyler Croy Baptiste Mathus Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit
[JIRA] (JENKINS-49406) Prototype the Evergreen snapshotting data safety system
Title: Message Title
Sam Gleske edited a comment on JENKINS-49406
Re: Prototype the Evergreen snapshotting data safety system
Example from my Jenkins RPM package: * [preUninstall.sh script running dailycommit.sh to save a copy of configuration before package upgrade|https://github.com/samrocketman/jenkins-bootstrap-shared/blob/0aa6a14868bb05105bb558c9c4d9ba0bca8a9d04/packaging/preUninstall.sh.in].* [Example gitignore used for my JENKINS_HOME|https://github.com/samrocketman/jenkins-bootstrap-shared/blob/0aa6a14868bb05105bb558c9c4d9ba0bca8a9d04/packaging/share/gitignore.in].* [Contents of dailycommit.sh|https://github.com/samrocketman/jenkins-bootstrap-shared/blob/0aa6a14868bb05105bb558c9c4d9ba0bca8a9d04/packaging/share/dailycommit.sh.in].That repository supports packaging Jenkins and plugins into multiple formats.{noformat}./gradlew buildRpm./gradlew buildDeb./gradlew buildTar#or package all three with ./gradlew packages#docker requires buildTardocker build -t jenkins .{noformat} h1. Additional notes * One of the challenges I discussed with [~rtyler] was setting workspaces for jobs building on master outside of JENKINS_HOME. Otherwise, you encounter weird issues with Git repositories inside of other Git repositories when they're not submodules. In general, we know it's bad practice for people to build on the master but it still gets done.* The gitignore file I linked intentionally does not track secret.key or the secrets directory. The intention here is that secrets get backed up separately from the encrypted configuration. However, this may not matter to some organizations. * Eventually, I want to completely rewrite the service scripts I copied from jenkins-packaging. Mainly because I have a different style of bash writing and will propose my changes back.
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
[JIRA] (JENKINS-49406) Prototype the Evergreen snapshotting data safety system
Title: Message Title
Sam Gleske edited a comment on JENKINS-49406
Re: Prototype the Evergreen snapshotting data safety system
Example from my Jenkins RPM package:* [preUninstall.sh script running dailycommit.sh to save a copy of configuration before package upgrade|https://github.com/samrocketman/jenkins-bootstrap-shared/blob/0aa6a14868bb05105bb558c9c4d9ba0bca8a9d04/packaging/preUninstall.sh.in].* [Example gitignore used for my JENKINS_HOME|https://github.com/samrocketman/jenkins-bootstrap-shared/blob/0aa6a14868bb05105bb558c9c4d9ba0bca8a9d04/packaging/share/gitignore.in].* [Contents of dailycommit.sh|https://github.com/samrocketman/jenkins-bootstrap-shared/blob/0aa6a14868bb05105bb558c9c4d9ba0bca8a9d04/packaging/share/dailycommit.sh.in].That repository supports packaging Jenkins and plugins into multiple formats.{noformat}./gradlew buildRpm./gradlew buildDeb./gradlew buildTar#or package all three with ./gradlew packages#docker requires buildTardocker build -t jenkins .{noformat} h1. Additional notes* One of the challenges I discussed with [~rtyler] was setting workspaces for jobs building on master outside of JENKINS_HOME. Otherwise, you encounter weird issues with Git repositories inside of other Git repositories when they're not submodules. In general, we know it's bad practice for people to build on the master but it still gets done.* The gitignore file I linked intentionally does not track secret.key or the secrets directory. The intention here is that secrets get backed up separately from the encrypted configuration. However, this may not matter to some organizations.
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
[JIRA] (JENKINS-49406) Prototype the Evergreen snapshotting data safety system
Title: Message Title Sam Gleske commented on JENKINS-49406 Re: Prototype the Evergreen snapshotting data safety system Example from my Jenkins RPM package: preUninstall.sh script running dailycommit.sh to save a copy of configuration before package upgrade. Example gitignore used for my JENKINS_HOME. Contents of dailycommit.sh. That repository supports packaging Jenkins and plugins into multiple formats. ./gradlew buildRpm ./gradlew buildDeb ./gradlew buildTar #or package all three with ./gradlew packages #docker requires buildTar docker build -t jenkins . Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
