[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Christoffer Cortes Sjöwall commented on JENKINS-52636 Re: Gerrit triggered jobs getting delayed Recently 2.30.0 was released that include two changes meant to reduce queue load. One reduces the disk writing when playback is enabled and the other gives you the ability to filter out unnecessary gerrit messages from the main settings panel under advanced. Some may still be experiencing delays and queue build ups though. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.192488.1531933057000.8563.1565081580963%40Atlassian.JIRA.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Jia Jia commented on JENKINS-52636 Re: Gerrit triggered jobs getting delayed Any status on this? There's also a chance that my team just encountered this recently. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.192488.1531933057000.8293.1565061540744%40Atlassian.JIRA.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Dustin Oprea edited a comment on JENKINS-52636 Re: Gerrit triggered jobs getting delayed We tried bumping to 2.29.0 during a system upgrade, and the queue started wildly building accumulating without actually starting any builds . We reverted the plugin to 2.27.5 and things appear to be rolling again. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.192488.1531933057000.2259.1564256580940%40Atlassian.JIRA.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Dustin Oprea commented on JENKINS-52636 Re: Gerrit triggered jobs getting delayed We tried bumping to 2.29.0 during a system upgrade, and the queue started wildly building. We reverted the plugin to 2.27.5 and things appear to be rolling again. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.192488.1531933057000.2237.1564255440420%40Atlassian.JIRA.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Darrien Glasser commented on JENKINS-52636 Re: Gerrit triggered jobs getting delayed It doesn't look like there's been any action on the PR yet although it mostly looks good to go from here. In the meantime, I have a patched version we're using at the company I'm at. You're free to use while we wait: https://github.com/DarrienG/gerrit-trigger-plugin/releases/tag/2.31.0-uninterested It's basically HEAD from the official repo + ignores irrelevant events. We have probably 3000+ builds a day and haven't seen any issues with it while we wait. Jenkins was unusable for us otherwise. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.192488.1531933057000.2215.1564254180518%40Atlassian.JIRA.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Dustin Oprea commented on JENKINS-52636 Re: Gerrit triggered jobs getting delayed Any status on this? There's a chance that my team just encountered this. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.192488.1531933057000.2151.1564253280571%40Atlassian.JIRA.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Ganesh Saraf commented on JENKINS-52636 Re: Gerrit triggered jobs getting delayed Hi Christoffer, That is good news. When we can expect it to get released? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.192488.1531933057000.828.1557909960576%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Christoffer Cortes Sjöwall commented on JENKINS-52636 Re: Gerrit triggered jobs getting delayed I have to PRs related to this issue. https://github.com/jenkinsci/gerrit-trigger-plugin/pull/397 https://github.com/jenkinsci/gerrit-trigger-plugin/pull/398 The first one is to add an event filter to the gerrit event stream and the second one is delegate disk writing to a thread instead of letting the workers have a hold up. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Christoffer Cortes Sjöwall edited a comment on JENKINS-52636 Re: Gerrit triggered jobs getting delayed I have to two PRs related to this issue.[https://github.com/jenkinsci/gerrit-trigger-plugin/pull/397][https://github.com/jenkinsci/gerrit-trigger-plugin/pull/398]The first one is to add an event filter to the gerrit event stream and the second one is delegate disk writing to a thread instead of letting the workers have a hold up. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Darrien Glasser edited a comment on JENKINS-52636 Re: Gerrit triggered jobs getting delayed Our team has confirmed that Gerrit trigger listens to all gerrit events, including ones it cannot act upon. It takes an inordinate time to determine whether or not it is able to act on them, finally only throwing a `NoSuchMethodException` trying to reflect into an object and see if the event has what it is looking for.This wasn't really an issue before, as all data was stored in SQL somewhere. NoteDB stores all data (events included) in refs/changes/* which all generate events that Jenkins listens for.We started using the replication plugin, and added reviewers automatically when a review was posted, and our Jenkins server was inundated almost immediately.As it stands, this makes the gerrit-trigger plugin unusable for anybody using the replication plugin, and as gerrit generates more metadata over time, will make the plugin unusable in all general cases unless it starts only listening to relevant events. We've found this gerrit trigger implemented with pipelines does not have the issue [https://github.com/jenkinsci/gerrit-code-review-plugin]And will likely be switching to it in the future. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Darrien Glasser commented on JENKINS-52636 Re: Gerrit triggered jobs getting delayed Our team has confirmed that Gerrit trigger listens to all gerrit events, including ones it cannot act upon. It takes an inordinate time to determine whether or not it is able to act on them, finally only throwing a `NoSuchMethodException` trying to reflect into an object and see if the event has what it is looking for. This wasn't really an issue before, as all data was stored in SQL somewhere. NoteDB stores all data (events included) in refs/changes/* which all generate events that Jenkins listens for. We started using the replication plugin, and added reviewers automatically when a review was posted, and our Jenkins server was inundated almost immediately. As it stands, this makes the gerrit-trigger plugin unusable for anybody using the replication plugin, and as gerrit generates more metadata over time, will make the plugin unusable in all general cases unless it starts only listening to relevant events. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Darrien Glasser edited a comment on JENKINS-52636 Re: Gerrit triggered jobs getting delayed Update, we've been debugging this issue, and it seems like the Replication Plugin may be (somehow) interfering with the missed events playback plugin/event trigger.We're not entirely certain, but it almost looks like the event trigger (that sends builds to Jenkins) is receiving replication events and gets stuck trying to understand what to do with them.We will continue investigating. Disabling the replication plugin reduces our backlog queue to 0 and events are perfectly streamed to Jenkins. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Darrien Glasser commented on JENKINS-52636 Re: Gerrit triggered jobs getting delayed Update, we've been debugging this issue, and it seems like the Replication Plugin may be (somehow) interfering with the missed events playback plugin/event trigger. We're not entirely certain, but it almost looks like the event trigger (that sends builds to Jenkins) is receiving replication events and gets stuck trying to understand what to do with them. We will continue investigating. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Darrien Glasser commented on JENKINS-52636 Re: Gerrit triggered jobs getting delayed We're having the same issue. Running on a machine with 32 cores we have a ton of events queued up and the trigger is barely able to keep up with them. Sometimes Jenkins appears to be dropping builds because of this and some of the requests are not honored. Our updated thread pool size doesn't appear to be taking effect (although that should be put in a different issues). Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Marco Miller edited a comment on JENKINS-52636 Re: Gerrit triggered jobs getting delayed Thx [~scoheb]! ;)Btw, a colleague of ours noticed [1] recently, if need be, or there might be other /related PRs too:[1] [https://github.com/jenkinsci/gerrit-trigger-plugin/pull/363|https://github.com/jenkinsci/gerrit-trigger-plugin/pull/363/commits] Note: not sure how close [1] is to [~hugares]' well-detailed findings above, though (did not check myself). Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Marco Miller commented on JENKINS-52636 Re: Gerrit triggered jobs getting delayed Thx Scott Hebert! Btw, a colleague of ours noticed [1] recently, if need be, or there might be other /related PRs too: [1] https://github.com/jenkinsci/gerrit-trigger-plugin/pull/363 Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Scott Hebert assigned an issue to Scott Hebert Jenkins / JENKINS-52636 Gerrit triggered jobs getting delayed Change By: Scott Hebert Assignee: rsandell Scott Hebert Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Scott Hebert commented on JENKINS-52636 Re: Gerrit triggered jobs getting delayed Hugo Ares Fred May Ganesh Saraf I will try to look into this shortly. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Ganesh Saraf commented on JENKINS-52636 Re: Gerrit triggered jobs getting delayed Scott Hebert , Can you please let us know by any change you might have looked into this? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Fred May commented on JENKINS-52636 Re: Gerrit triggered jobs getting delayed In the situation above, we have found the vast majority of the Gerrit events relate to "git notes" in the git repos. They are not required for triggering Jenkins jobs. A plugin config option that would allow certain events to be ignored by the plugin would be useful. Add Comment This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Amit Sharma updated an issue Jenkins / JENKINS-52636 Gerrit triggered jobs getting delayed Change By: Amit Sharma Priority: Minor Critical Add Comment This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Hugo Ares updated an issue Jenkins / JENKINS-52636 Gerrit triggered jobs getting delayed Change By: Hugo Ares Component/s: pipeline Component/s: script-security-plugin Add Comment This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Hugo Ares commented on JENKINS-52636 Re: Gerrit triggered jobs getting delayed Scott Hebert Any chance you have time to fix this Add Comment This message was sent by Atlassian JIRA (v7.10.1#710002-sha1:6efc396) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-52636) Gerrit triggered jobs getting delayed
Title: Message Title Hugo Ares commented on JENKINS-52636 Re: Gerrit triggered jobs getting delayed I investigated the issue as I am experiencing it in my own jenkins server and here are my findings. The issue only happens when Gerrit-Trigger is connected to a Gerrit server with a lot of activity(events). Even if Gerrit-Trigger uses a thread pool(with 3 threads by default) to process events, it cannot keep up, the queue of events to process grows until the build up causes delays in triggering the builds(can be few hours). Cranking up the number of threads does not really help as there is a major bottleneck in the Gerrit-Trigger code. The bottleneck is in GerritMissedEventsPlaybackManager.persist method. This method serialize to disk the last event received in order for the playback manager to know what is the last event processed and retrigger missed event while Gerrit-Trigger was down in case it goes down or the connection towards Gerrit is lost. The problem is that method is synchronized and called by the threads processing the events. So even if you have 100 receiving threads, they will all bottleneck in that method which is very slow as it does IO, this is even worst if Jenkins home is on NFS or any kind of slow volume. The stack trace in the description is not right, here is the stack traces of a blocked thread followed by the stack trace of the thread holding the lock: "Gerrit Worker EventThread_0" Id=62 Group=main BLOCKED on com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager@24996aa8 owned by "Gerrit Worker EventThread_1" Id=63 at com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager.persist(GerritMissedEventsPlaybackManager.java:455) - blocked on com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager@24996aa8 at com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager.gerritEvent(GerritMissedEventsPlaybackManager.java:300) at com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListener(GerritHandler.java:350) at com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListeners(GerritHandler.java:317) at com.sonyericsson.hudson.plugins.gerrit.trigger.JenkinsAwareGerritHandler.notifyListeners(JenkinsAwareGerritHandler.java:77) at com.sonymobile.tools.gerrit.gerritevents.workers.AbstractGerritEventWork.perform(AbstractGerritEventWork.java:46) at com.sonymobile.tools.gerrit.gerritevents.workers.AbstractJsonObjectWork.perform(AbstractJsonObjectWork.java:77) at com.sonymobile.tools.gerrit.gerritevents.workers.StreamEventsStringWork.perform(StreamEventsStringWork.java:67) at com.sonymobile.tools.gerrit.gerritevents.workers.EventThread.run(EventThread.java:66) at com.sonyericsson.hudson.plugins.gerrit.trigger.SystemEventThread.run(SystemEventThread.java:66) "Gerrit Worker EventThread_1" Id=63 Group=main RUNNABLE at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromItemType(MapperWrapper.java:88) at com.thoughtworks.xstream.mapper.MapperWrapper.getConverterFromItemType(MapperWrapper.java:88) at
