[JIRA] (JENKINS-38381) Optimize log handling in Pipeline and Durable Task
Title: Message Title Jenn Briden updated an issue Jenkins / JENKINS-38381 Optimize log handling in Pipeline and Durable Task Change By: Jenn Briden Labels: blocked cloudbees-internal-pipeline performance project-cheetah Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38381) Optimize log handling in Pipeline and Durable Task
Title: Message Title Vivek Pandey updated an issue Jenkins / JENKINS-38381 Optimize log handling in Pipeline and Durable Task Change By: Vivek Pandey Sprint: Pipeline - October, Pipeline - December March 2018 Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38381) Optimize log handling in Pipeline and Durable Task
Title: Message Title mishal shah commented on JENKINS-38381 Re: Optimize log handling in Pipeline and Durable Task Jesse Glick Do you know when this will be released? We are also seeing infinite log issue. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38381) Optimize log handling in Pipeline and Durable Task
Title: Message Title Sam Van Oort updated an issue Jenkins / JENKINS-38381 Optimize log handling in Pipeline and Durable Task Change By: Sam Van Oort Labels: cloudbees-internal-pipeline performance project-cheetah Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38381) Optimize log handling in Pipeline and Durable Task
Title: Message Title Jesse Glick commented on JENKINS-38381 Re: Optimize log handling in Pipeline and Durable Task workflow-durable-task-step #65 covers everything I have updated recently: there is now a set of independently mergeable PRs for: push rather than pull of durable task output proper handling of Unicode (JENKINS-31096) both of those together (so they could be merged in either order) workflow-job #27 is now downstream of those changes, and continues to encapsulate the broader rewrite: single log sink for all Pipeline steps SPI to allow that sink be an external service redesign of step console annotations for a more usable display in the classic UI (this requires the first item) Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38381) Optimize log handling in Pipeline and Durable Task
Title: Message Title Jesse Glick commented on JENKINS-38381 Re: Optimize log handling in Pipeline and Durable Task Did some optimizations; down to: Took 2525ms to run the build Took 761ms to write HTML of whole build Took 133ms to compute length of whole build Took 86ms to write truncated HTML of whole build Took 111ms to write plain text of whole build Took 379ms to write HTML of one node Took 57ms to compute length of one node Took 64ms to write truncated HTML of one node Took 62ms to write plain text of one node Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38381) Optimize log handling in Pipeline and Durable Task
Title: Message Title Jesse Glick commented on JENKINS-38381 Re: Optimize log handling in Pipeline and Durable Task Baseline (pre-patch) performance for a build with one (echo) step producing a million lines of output on my laptop: Took 1791ms to run the build Took 437ms to write HTML of whole build Took 0ms to compute length of whole build Took 19ms to write truncated HTML of whole build Took 11ms to write plain text of whole build Took 340ms to write HTML of one node Took 0ms to compute length of one node Took 6ms to write truncated HTML of one node Took 10ms to write plain text of one node With the patch (but not using remote log storage): Took 3687ms to run the build Took 866ms to write HTML of whole build Took 127ms to compute length of whole build Took 129ms to write truncated HTML of whole build Took 162ms to write plain text of whole build Took 630ms to write HTML of one node Took 115ms to compute length of one node Took 148ms to write truncated HTML of one node Took 75ms to write plain text of one node The build itself seems to take longer, suggesting the need for optimization in AnnotatedLogAction.DecoratedTaskListener. (Though this is not a very realistic case, since it represents a step producing megabytes of output all at once, making things easy on copyLogs; in practice output would come incrementally, which the baseline version would probably spend more time copying.) As to the rendering of the log (if and when someone asks for it), the second set of numbers represent measurable load on the server, but a log this big seems like it would be rare in the field, and anyway the cost of rendering such log output in Chrome on the client side is much higher. In other words, the subjective overhead of viewing the build is mostly due to the browser, not any processing on the server. time curl -I … produces responses rather quickly. So I am less concerned about that.
[JIRA] (JENKINS-38381) Optimize log handling in Pipeline and Durable Task
Title: Message Title recampbell edited a comment on JENKINS-38381 Re: Optimize log handling in Pipeline and Durable Task Benefits of this change: This change will fix unicode support. This change will also send all log information to a single sink instead of seperate logfiles per step, removing a bottleneck in the build logging. Also, instead of the master polling for output from the pipeline, the logs are instead pushed from the slave to the master.We agree that we should implement this change, but leave the extension point ( ` PipelineLogFile ) ` marked as beta.Our plan is for Jesse to complete testing of this change and integrate it into the master branch when ready. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38381) Optimize log handling in Pipeline and Durable Task
Title: Message Title recampbell commented on JENKINS-38381 Re: Optimize log handling in Pipeline and Durable Task Benefits of this change: This change will fix unicode support. This change will also send all log information to a single sink instead of seperate logfiles per step, removing a bottleneck in the build logging. Also, instead of the master polling for output from the pipeline, the logs are instead pushed from the slave to the master. We agree that we should implement this change, but leave the extension point (PipelineLogFile) marked as beta. Our plan is for Jesse to complete testing of this change and integrate it into the master branch when ready. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38381) Optimize log handling in Pipeline and Durable Task
Title: Message Title Jesse Glick commented on JENKINS-38381 Re: Optimize log handling in Pipeline and Durable Task Integrated demo is in the logstash PR. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38381) Optimize log handling in Pipeline and Durable Task
Title: Message Title Jesse Glick updated JENKINS-38381 Jenkins / JENKINS-38381 Optimize log handling in Pipeline and Durable Task Change By: Jesse Glick Status: In Progress Review Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38381) Optimize log handling in Pipeline and Durable Task
Title: Message Title SCM/JIRA link daemon commented on JENKINS-38381 Re: Optimize log handling in Pipeline and Durable Task Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/org/jenkinsci/plugins/durabletask/BourneShellScript.java src/main/java/org/jenkinsci/plugins/durabletask/Controller.java src/main/java/org/jenkinsci/plugins/durabletask/DurableTask.java src/main/java/org/jenkinsci/plugins/durabletask/FileMonitoringTask.java src/main/java/org/jenkinsci/plugins/durabletask/Handler.java src/main/java/org/jenkinsci/plugins/durabletask/executors/OnceRetentionStrategy.java src/test/java/org/jenkinsci/plugins/durabletask/BourneShellScriptTest.java http://jenkins-ci.org/commit/durable-task-plugin/eaa6b02e8a2aec369f727c29a15a7fc44ebcd12a Log: JENKINS-38381 Prototype API to receive asynchronous notifications of process output or exit code. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38381) Optimize log handling in Pipeline and Durable Task
Title: Message Title Jesse Glick commented on JENKINS-38381 Re: Optimize log handling in Pipeline and Durable Task Basic stuff now seems to be working. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-38381) Optimize log handling in Pipeline and Durable Task
Title: Message Title Oleg Nenashev updated an issue Jenkins / JENKINS-38381 Optimize log handling in Pipeline and Durable Task Change By: Oleg Nenashev Summary: Optimize log handling in Pipeline and Durable Task Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
