[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title Vivek Pandey updated JENKINS-38335 Jenkins / JENKINS-38335 Search API too slow for dashboard Change By: Vivek Pandey Status: In Review Resolved Resolution: Fixed Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title Michael Neale commented on JENKINS-38335 Re: Search API too slow for dashboard YAY, looks like this solves a lot... Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title Vivek Pandey updated JENKINS-38335 Jenkins / JENKINS-38335 Search API too slow for dashboard Change By: Vivek Pandey Status: In Progress Review Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title Vivek Pandey commented on JENKINS-38335 Re: Search API too slow for dashboard Looks like computing numberOfRunningPipelines is causing this extremely slow behavior. Below is my analysis: Running following GET call takes anywhere from 13 sec to 30sec or sometimes 504 error. GET https://JENKINS_HOST/blue/rest/search/?q=type:pipeline;excludedFromFlattening:jenkins.branch.MultiBranchProject,hudson.matrix.MatrixProject=no-folders=0=26=class,name,_links,displayName,fullName,permissions,numberOfQueuedPipelines,organization,branchNames,latestRun,numberOfRunningPipelines or GET https://JENKINS_HOST/blue/rest/search/?q=type:pipeline;excludedFromFlattening:jenkins.branch.MultiBranchProject,hudson.matrix.MatrixProject=no-folders=0=26=numberOfRunningPipelines Running query below takes 148 ms. Notice there is no numberOfRunningPipelines element serialized. Clearly computation of this in case of multi-branch pipeline means going over all builds of each branch and then filter running builds. I will get back on this after more investigation. GET https://JENKINS_HOST/blue/rest/search/?q=type:pipeline;excludedFromFlattening:jenkins.branch.MultiBranchProject,hudson.matrix.MatrixProject=no-folders=0=26=class,name,_links,displayName,fullName,permissions,numberOfQueuedPipelines,organization,branchNames,latestRun I noticed that UI is not even using numberOfRunningPipelines and there is no optimal way to compute this number so I am going to remove it from backend. I will submit a PR where will get UI team to take a look at it. Add Comment
[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title Vivek Pandey assigned an issue to Vivek Pandey Jenkins / JENKINS-38335 Search API too slow for dashboard Change By: Vivek Pandey Assignee: Keith Zantow Vivek Pandey Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title Michael Neale updated an issue Jenkins / JENKINS-38335 Search API too slow for dashboard Change By: Michael Neale Sprint: pacific, atlantic, 1.0-b05/b-06 Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title James Dumay updated an issue Jenkins / JENKINS-38335 Search API too slow for dashboard Change By: James Dumay Sprint: 26-september, 1.0-b05/b-06 Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title Michael Neale commented on JENKINS-38335 Re: Search API too slow for dashboard Punting this to keith for now as Vivek away. Looks like until we get a JENKINS_HOME that shows the search slowness, there are probably other avenues to explore to optimise. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title Michael Neale assigned an issue to Keith Zantow Jenkins / JENKINS-38335 Search API too slow for dashboard Change By: Michael Neale Assignee: Vivek Pandey Keith Zantow Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title Vivek Pandey commented on JENKINS-38335 Re: Search API too slow for dashboard Keith Zantow My initial investigation report: PipelineSearch calls, Jenkins.getInstance().getAllItems(). Jenkins keeps all these items in memory so its all in memory cost PipelineSearch does few (at least 2 iteration) of all items and wraps them in BluePipeline implementation object Only pagination size limit, for example 26 BluePipeline items gets serialized over wire Most cost is paid during serialization Things like get build id, get last build, run details, actions attached to pipeline job, actions attached to jenkins.model.Run etc, where the real cost is. I tried ci.blueocean.io jobs locally (using downloaded jenkins home content of dogfood ci server) as well as the one running over ci.blueocean.io - but they do not indicate expensiv e search API calls. 95ms local, 364ms at ci.blueocean.io. See attached images. I tried profiling running instance blueocean locally with dogfood home directory, but the cost paid during fetching pipeline results was negligible. I suspect there are some plugins, who contribute to actions that contribute to the slow behavior so if Ben Walding could his work load that can help us identify why its slow on his instance. search-perf-dogfood-remote.tiff Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title Vivek Pandey updated an issue Jenkins / JENKINS-38335 Search API too slow for dashboard Change By: Vivek Pandey Attachment: search-perf-dogfood-local.tiff Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title Vivek Pandey edited a comment on JENKINS-38335 Re: Search API too slow for dashboard [~kzantow] My initial investigation report:* PipelineSearch calls, Jenkins.getInstance().getAllItems(). Jenkins keeps all these items in memory so its all in memory cost* PipelineSearch does few (at least 2 iteration) of all items and wraps them in BluePipeline implementation object* Only pagination size limit, for example 26 BluePipeline items gets serialized over wire* Most cost is paid during serialization** Things like get build id, get last build, run details, actions attached to pipeline job, actions attached to jenkins.model.Run etc, where the real cost is.* I tried ci.blueocean.io jobs locally (using downloaded jenkins home content of dogfood ci server) as well as the one running over ci.blueocean.io - but they do not indicate expensiv !search-perf-dogfood-remote.tiff|thumbnail! e search API calls. 95ms local, 364ms at ci.blueocean.io. See attached images.* I tried profiling running instance blueocean locally with dogfood home directory, but the cost paid during fetching pipeline results was negligible.* I suspect there are some plugins, who contribute to actions that contribute to the slow behavior so if [~bwalding] could his work load that can help us identify why its slow on his instance. ! [^ search-perf-dogfood- local remote .tiff |thumbnail! ] [^search-perf-dogfood- remote local .tiff] Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails
[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title Vivek Pandey updated an issue Jenkins / JENKINS-38335 Search API too slow for dashboard Change By: Vivek Pandey Attachment: search-perf-dogfood-remote.tiff Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title Michael Neale updated an issue Jenkins / JENKINS-38335 Search API too slow for dashboard Change By: Michael Neale *Search API (page limit 26)** Took 7 sec on average (in other cases, 12seconds for 11kb of data - from [~bwalding])* To address sorting by name and exclude certain items from flattening all possible items are loaded in memory** jenkins.getAllItems()** Exclude items that are children of multi-branch and matrix project** Don’t see how we can avoid it unless the jobs are stored in some kind of db with indexed column to speed up sorting and only give limited records instead of all* new PipelineContainerImpl().getPipelines(items) should be pagination aware* At present it creates BluePipeline object for all possible items (ouch!) Also: check that runs/activity benefits from same pagination aware optimisation Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues"
[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title Vivek Pandey commented on JENKINS-38335 Re: Search API too slow for dashboard Michael Neale https://issues.jenkins-ci.org/browse/JENKINS-38087 is duplicate, one of them should be closed. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title Vivek Pandey started work on JENKINS-38335 Change By: Vivek Pandey Status: Open In Progress Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title James Dumay assigned an issue to Vivek Pandey Jenkins / JENKINS-38335 Search API too slow for dashboard Change By: James Dumay Assignee: Vivek Pandey Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title Michael Neale updated an issue Jenkins / JENKINS-38335 Search API too slow for dashboard Change By: Michael Neale *Search API (page limit 26)** Took 7 sec on average (in other cases, 12seconds for 11kb of data - from [~bwalding]) * To address sorting by name and exclude certain items from flattening all possible items are loaded in memory** jenkins.getAllItems()** Exclude items that are children of multi-branch and matrix project** Don’t see how we can avoid it unless the jobs are stored in some kind of db with indexed column to speed up sorting and only give limited records instead of all* new PipelineContainerImpl().getPipelines(items) should be pagination aware* At present it creates BluePipeline object for all possible items (ouch!) Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an
[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title Michael Neale updated an issue Jenkins / JENKINS-38335 Search API too slow for dashboard Change By: Michael Neale Sprint: 1.0-b05/b-06 Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title Michael Neale updated an issue Jenkins / JENKINS-38335 Search API too slow for dashboard Change By: Michael Neale *Search API (page limit 26)** Took 7 sec on average* To address sorting by name and exclude certain items from flattening all possible items are loaded in memory* * jenkins.getAllItems()* * Exclude items that are children of multi-branch and matrix project* * Don’t see how we can avoid it unless the jobs are stored in some kind of db with indexed column to speed up sorting and only give limited records instead of all* new PipelineContainerImpl().getPipelines(items) should be pagination aware* At present it creates BluePipeline object for all possible items (ouch!) Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For
[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title Michael Neale commented on JENKINS-38335 Re: Search API too slow for dashboard ping James Dumay Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38335) Search API too slow for dashboard
Title: Message Title Michael Neale created an issue Jenkins / JENKINS-38335 Search API too slow for dashboard Issue Type: Bug Assignee: Unassigned Components: blueocean-plugin Created: 2016/Sep/19 6:32 AM Priority: Critical Reporter: Michael Neale Search API (page limit 26) Took 7 sec on average To address sorting by name and exclude certain items from flattening all possible items are loaded in memory jenkins.getAllItems() Exclude items that are children of multi-branch and matrix project Don’t see how we can avoid it unless the jobs are stored in some kind of db with indexed column to speed up sorting and only give limited records instead of all new PipelineContainerImpl().getPipelines(items) should be pagination aware At present it creates BluePipeline object for all possible items (ouch!)
