[JIRA] (JENKINS-13613) scm_sync_configuration may commit bulk changes in one change set
Wolfgang Hauser commented on JENKINS-13613 scm_sync_configuration may commit bulk changes in one change set I added some files and a directory to be included as user defined confirurations and afterwords I got following error while opening the Jenkons system configuration: Exception: javax.servlet.ServletException: org.apache.commons.jelly.JellyTagException: jar:file:/hudson/jenkins/jenkins/war/WEB-INF/lib/jenkins-core-1.482.jar!/jenkins/model/Jenkins/configure.jelly:56:86: j:forEach GC overhead limit exceeded Stacktrace: javax.servlet.ServletException: javax.servlet.ServletException: org.apache.commons.jelly.JellyTagException: jar:file:/hudson/jenkins/jenkins/war/WEB-INF/lib/jenkins-core-1.482.jar!/jenkins/model/Jenkins/configure.jelly:56:86: j:forEach GC overhead limit exceeded at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:44) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at
[JIRA] (JENKINS-13613) scm_sync_configuration may commit bulk changes in one change set
Frédéric Camblor commented on JENKINS-13613 scm_sync_configuration may commit bulk changes in one change set I don't think this is related to the plugin (it seems it is because of hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter in the stacktrace... but this is a false positive because in this case we are just in sort of a http filter). Please see this wiki entry This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira
[JIRA] (JENKINS-13613) scm_sync_configuration may commit bulk changes in one change set
Wolfgang Hauser commented on JENKINS-13613 scm_sync_configuration may commit bulk changes in one change set After a restart of Jenkins, the error does not occur anymore and the (bulk)commit works fine. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira
[JIRA] (JENKINS-13613) scm_sync_configuration may commit bulk changes in one change set
Frédéric Camblor commented on JENKINS-13613 scm_sync_configuration may commit bulk changes in one change set Cool This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira
[JIRA] (JENKINS-13613) scm_sync_configuration may commit bulk changes in one change set
Wolfgang Hauser commented on JENKINS-13613 scm_sync_configuration may commit bulk changes in one change set If I change my configuration on disk and reload configuration from disk, the changes are NOT recognized by the plugin. Same, if I restart Jenkins. I would expect a synchronization, but nothing is done. Should this be a new Issue, or should this Issue be reopened ? This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira
[JIRA] (JENKINS-13613) scm_sync_configuration may commit bulk changes in one change set
Frédéric Camblor commented on JENKINS-13613 scm_sync_configuration may commit bulk changes in one change set Yup this is not related to this issue. I forgot to test this before releasing, could you file an issue please ? This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira
[JIRA] (JENKINS-13613) scm_sync_configuration may commit bulk changes in one change set
Frédéric Camblor resolved JENKINS-13613 as Fixed scm_sync_configuration may commit bulk changes in one change set Transactional commit will be available in v0.0.6 I released today. Please, could you test it and provide any feedback about it ? Change By: Frédéric Camblor (18/Sep/12 12:04 AM) Status: InProgress Resolved Fix Version/s: current Resolution: Fixed This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira
[JIRA] (JENKINS-13613) scm_sync_configuration may commit bulk changes in one change set
Frédéric Camblor started work on JENKINS-13613 scm_sync_configuration may commit bulk changes in one change set Change By: Frédéric Camblor (25/Jul/12 8:38 AM) Status: Open InProgress This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira
[JIRA] (JENKINS-13613) scm_sync_configuration may commit bulk changes in one change set
Frédéric Camblor commented on JENKINS-13613 scm_sync_configuration may commit bulk changes in one change set I'm going to refactor the way the plugin is behaving concerning commits. For the moment, it detects a change on a Saveable object and, each time, make a commit on the file concerned by this change. You have to note that a Saveable is always attached to a file, but a file can be concerned by several Saveable (ie: descriptors) This is the reason why, when, for instance, you persist the jenkins system configuration screen, lots of commit will be made : every plugin which has a Descriptor on system configuration data, will be saved separately = each save will make a commit. This is the point of the issue. What I'm going to do to address this problem : I'll implement a PluginServletFilter that will initialize, for every requests, an empty Changeset in the beginning of the request. This Changeset will be stored in a ThreadLocal. Then, request will be processed normally Then, when request process ends, the PluginServletFilter will look at the Changeset and, if it isn't empy, it will commit every changes in this one, in only one commit. Now, some more technical inputs : Everytime a Saveable will be saved, instead of synchronizing the underlying file, current Saveable's file content will be added into the ThreadLocal's Changeset A Changeset will be made of a MapfilePath, contentOfFile. Thus, if multiple Saveable work in the same file, they will override same Changeset entry each time the Saveable is saved. I hope the byte[] won't consume too much memory... This Changeset representation will avoid collision on file save : 2 users edit the global jenkins configuration page 1st user submit the page, it takes a long time to synchronize... 2nd user submit the page, it modifies the config.xml before synchronization of 1st user is finished = changes of 2nd user will be commited during 1st user commit Commit could then be made asynchronously, but this is another point (see JENKINS-14214) I think by doing such a refactoring, I could fix problems like JENKINS-10967 and JENKINS-9166 This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira
[JIRA] (JENKINS-13613) scm_sync_configuration may commit bulk changes in one change set
Frédéric Camblor updated JENKINS-13613 scm_sync_configuration may commit bulk changes in one change set Change By: Frédéric Camblor (25/Jul/12 9:03 AM) Priority: Minor Major This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira
[JIRA] (JENKINS-13613) scm_sync_configuration may commit bulk changes in one change set
Frédéric Camblor edited a comment on JENKINS-13613 scm_sync_configuration may commit bulk changes in one change set I'm going to refactor the way the plugin is behaving concerning commits. For the moment, it detects a change on a Saveable object and, each time, make a commit on the file concerned by this change. You have to note that a Saveable is always attached to a file, but a file can be concerned by several Saveable (ie: descriptors) This is the reason why, when, for instance, you persist the jenkins system configuration screen, lots of commit will be made : every plugin which has a Descriptor on system configuration data, will be saved separately = each save will make a commit. This is the point of the issue. What I'm going to do to address this problem : I'll implement a PluginServletFilter that will initialize, for every requests, an empty Changeset in the beginning of the request. This Changeset will be stored in a ThreadLocal. Then, request will be processed normally Then, when request process ends, the PluginServletFilter will look at the Changeset and, if it isn't empy, it will commit every changes in this one, in only one commit. Now, some more technical inputs : Everytime a Saveable will be saved, instead of synchronizing the underlying file, current Saveable's file content will be added into the ThreadLocal's Changeset A Changeset will be made of a MapfilePath, contentOfFile. Thus, if multiple Saveable work in the same file, they will override same Changeset entry each time the Saveable is saved. I hope the byte[] won't consume too much memory... This Changeset representation will avoid collision on file save : 2 users edit the global jenkins configuration page 1st user submit the page, it takes a long time to synchronize... 2nd user submit the page, it modifies the config.xml before synchronization of 1st user is finished = changes of 2nd user will be commited during 1st user commit Commit could then be made asynchronously, but this is another point (see JENKINS-14214) When Saveable.save() is called "outside" an http request (triggerd by jenkins himself for instance), I won't be able to identify (for the moment) clearly a "transaction". Thus, I'll have to keep the "pre" JENKINS-13613 behaviour as a fallback : 1 commit per Saveable.save(). On this point, though, maybe could I ask Kohsuke if he has a good entry point for batched jenkins task when I could say "hey, transaction starts here, start to track changesets on Saveable.save()" I think by doing such a refactoring, I could fix problems like JENKINS-10967 and JENKINS-9166 This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira
[JIRA] (JENKINS-13613) scm_sync_configuration may commit bulk changes in one change set
Wolfgang Hauser created JENKINS-13613: - Summary: scm_sync_configuration may commit bulk changes in one change set Key: JENKINS-13613 URL: https://issues.jenkins-ci.org/browse/JENKINS-13613 Project: Jenkins Issue Type: Improvement Components: scm-sync-configuration Affects Versions: current Reporter: Wolfgang Hauser Assignee: Frédéric Camblor Priority: Minor Fix For: current It would be good, if bulk changes are committed in one change set. Currently all changed files are committed in single steps. If changes are done off line, it may be better to have one change set for all involved changes to be able to roll back it with a single scm command. This may only occur if the plugin collect the changes at startup and reload configuration from hard disk. Normal changes from UI should be sent immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
