[JIRA] (JENKINS-30896) Unoptimized node & log storage
Title: Message Title Jesse Glick commented on JENKINS-30896 Re: Unoptimized node & log storage Splitting log portion into JENKINS-38381 since I think it is independent. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-30896) Unoptimized node & log storage
Title: Message Title Sam Van Oort commented on JENKINS-30896 Re: Unoptimized node & log storage Note for later implementation of log consolidation: WorkflowRun.copyLogs + logNodeMessage (writing to a WorkflowConsoleLogger) IIUC there's a StreamBuildListener attached to the WorkflowRun which gets log outputs, this is sliced and copied into a new log file for each FlowNode periodically (at the least upon completion, via copyLogs), which becomes part of a LogActionImpl. If we wanted we could write directly to an output file, and then use an alternate LogAction version that gives offsets within this file. Interleaved streams from parallels are an issue though (probably we need to keep separate log streams for each, numbered by parallel # or iota and branch order, I.E. 1-1.log, 1-2.log, 2-1.log, 2-2.log). Would need to create new StreamBuildListeners in this case. Possible case also for something like NIO w/ channels (since we're basically redirecting inputs from one channel into multiplexing (reactor pattern?). Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-30896) Unoptimized node & log storage
Title: Message Title Jesse Glick updated an issue Jenkins / JENKINS-30896 Unoptimized node & log storage Change By: Jesse Glick Component/s: workflow-support-plugin Component/s: pipeline Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-30896) Unoptimized node & log storage
Title: Message Title Sam Van Oort commented on JENKINS-30896 Re: Unoptimized node & log storage To give one order-of-magnitude estimate for scaling, for the CloudBees internal CI, mean and median iota across the last successful run of each pipeline job are 86 & 52 respectively. Runs with several hundred nodes are not uncommon. As long as we take some measures to simplify serialization/deserialization in storage to reduce the CPU hit, batching nodes in chunks of 100 would be reasonable (1-2 kB per node, an entire flow can be loaded/saved in one 50-200 kB I/O). Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
