[jira] [Commented] (TS-1521) Enable compression for binary log format
[ https://issues.apache.org/jira/browse/TS-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421613#comment-15421613 ] Bryan Call commented on TS-1521: If this is something you want to work on please submit a github patch. If not please close. > Enable compression for binary log format > > > Key: TS-1521 > URL: https://issues.apache.org/jira/browse/TS-1521 > Project: Traffic Server > Issue Type: New Feature > Components: Logging > Environment: RHEL 6+ >Reporter: Lans Carstensen > Fix For: sometime > > Attachments: logcompress.patch > > > As noted by in a discussion on #traffic-server, gzip can result in 90%+ > compression on the binary access logs. By adding a reasonable streaming > compression algorithm to the binary format you could significantly reduce > logging-related IOPS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-1521) Enable compression for binary log format
[ https://issues.apache.org/jira/browse/TS-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13919388#comment-13919388 ] Zhao Yongming commented on TS-1521: --- [~bettydreamit] submit thire gzipping patch for the ascii loging, please consider accept this feature too. Enable compression for binary log format Key: TS-1521 URL: https://issues.apache.org/jira/browse/TS-1521 Project: Traffic Server Issue Type: New Feature Components: Logging Environment: RHEL 6+ Reporter: Lans Carstensen Assignee: Yunkai Zhang Fix For: 6.0.0 Attachments: logcompress.patch As noted by in a discussion on #traffic-server, gzip can result in 90%+ compression on the binary access logs. By adding a reasonable streaming compression algorithm to the binary format you could significantly reduce logging-related IOPS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-1521) Enable compression for binary log format
[ https://issues.apache.org/jira/browse/TS-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13919547#comment-13919547 ] Leif Hedstrom commented on TS-1521: --- Couple of thoughts: 1. This needs to be configurable, per log object. 2. Is this for ascii logs only? If so, why ? 3. There's a delete[] added in the patch, why is that ? Presumably there's another delete[] then that's superfluous ?? 4. The code uses tab's. Don't do that :). 5. Who is reggie ? Enable compression for binary log format Key: TS-1521 URL: https://issues.apache.org/jira/browse/TS-1521 Project: Traffic Server Issue Type: New Feature Components: Logging Environment: RHEL 6+ Reporter: Lans Carstensen Assignee: Yunkai Zhang Fix For: 6.0.0 Attachments: logcompress.patch As noted by in a discussion on #traffic-server, gzip can result in 90%+ compression on the binary access logs. By adding a reasonable streaming compression algorithm to the binary format you could significantly reduce logging-related IOPS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TS-1521) Enable compression for binary log format
[ https://issues.apache.org/jira/browse/TS-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777335#comment-13777335 ] Zhao Yongming commented on TS-1521: --- questions on how to implement: 1, should we enable gzip on client side? can we gzip on the collation side before farwording? in case the collation server is in the localnetwork and cache server heavy load 2, should we gzip in ET_NET threads? is ET_TASK a good option? 3, can we forward the gziped log on collation server? Enable compression for binary log format Key: TS-1521 URL: https://issues.apache.org/jira/browse/TS-1521 Project: Traffic Server Issue Type: Improvement Components: Logging Environment: RHEL 6+ Reporter: Lans Carstensen Assignee: Yunkai Zhang Fix For: 5.0.0 As noted by in a discussion on #traffic-server, gzip can result in 90%+ compression on the binary access logs. By adding a reasonable streaming compression algorithm to the binary format you could significantly reduce logging-related IOPS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-1521) Enable compression for binary log format
[ https://issues.apache.org/jira/browse/TS-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13777677#comment-13777677 ] Leif Hedstrom commented on TS-1521: --- Good questions. It certainly makes sense to gzip it before going on the wire IMO. It would also make sense to do that such that each log buffer (8KB by default, right?) gets compressed before either writing to disk, or sending on the wire. Remember that each log buffer can have different formatting under the hood, so it's important to preserve the exact buffer format upon unzip. I think this should run on ET_TASK, or perhaps the log thread(s) itself. I think we should avoid it on the ET_NET threads if possible, just like the compression for RAM cache did. We also should allow for inexpensive, and less efficient, compression options. Look at the compression feature for the RAM cache, it has a few options, and it would make sense to use (and perhaps refactor / generalize) that compression processing (is it a Processor now?). If we do, that would imply running it on the ET_TASK threads, which I'm fine with (we can easily control the number of ET_TASK threads). Finally, there is the beginning of a Processor on ET_TASK, so adding a gzip processing feature there would make sense. And, lets start writing up the design ideas and implementation details on the Projects area of the wiki :). Exciting! Enable compression for binary log format Key: TS-1521 URL: https://issues.apache.org/jira/browse/TS-1521 Project: Traffic Server Issue Type: New Feature Components: Logging Environment: RHEL 6+ Reporter: Lans Carstensen Assignee: Yunkai Zhang Fix For: 5.0.0 As noted by in a discussion on #traffic-server, gzip can result in 90%+ compression on the binary access logs. By adding a reasonable streaming compression algorithm to the binary format you could significantly reduce logging-related IOPS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-1521) Enable compression for binary log format
[ https://issues.apache.org/jira/browse/TS-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13695294#comment-13695294 ] Igor Galić commented on TS-1521: it may be generally an advantage to have streaming compression over all types of logs, not just binary logs. Binary logs take (our) special programs to interpret, whereas plain text can be interpreted by anyone - the gain from streaming compression for logs may give it a performance boost re disk write operations. Enable compression for binary log format Key: TS-1521 URL: https://issues.apache.org/jira/browse/TS-1521 Project: Traffic Server Issue Type: Improvement Components: Logging Environment: RHEL 6+ Reporter: Lans Carstensen Fix For: 3.5.0 As noted by in a discussion on #traffic-server, gzip can result in 90%+ compression on the binary access logs. By adding a reasonable streaming compression algorithm to the binary format you could significantly reduce logging-related IOPS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (TS-1521) Enable compression for binary log format
[ https://issues.apache.org/jira/browse/TS-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13475191#comment-13475191 ] Sean Kamath commented on TS-1521: - It would also be very beneficial when using a remote logging host to compress the stream to the other host. In fact, this would be prefereable to on-disk only compression, since sending the cluster logs to an aggregator can offload the disk writes. Enable compression for binary log format Key: TS-1521 URL: https://issues.apache.org/jira/browse/TS-1521 Project: Traffic Server Issue Type: Improvement Components: Logging Environment: RHEL 6+ Reporter: Lans Carstensen As noted by in a discussion on #traffic-server, gzip can result in 90%+ compression on the binary access logs. By adding a reasonable streaming compression algorithm to the binary format you could significantly reduce logging-related IOPS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
