[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095132#comment-15095132 ] ASF subversion and git services commented on TS-3072: - Commit 45266942b4510f5d06f2af95ea7b8972134ad8f8 in trafficserver's branch refs/heads/master from shinrich [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=4526694 ] TS-3072: Debug logging for single connection. Without config control. This closes #398. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072-config.diff, ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095362#comment-15095362 ] ASF subversion and git services commented on TS-3072: - Commit 51faf6ee98c351a2f1b545183e463e4fcde1d38f in trafficserver's branch refs/heads/6.1.x from shinrich [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=51faf6e ] TS-3072: fix newer compiler warnings. (cherry picked from commit 7cd050790b4cc6a85becebef7438e01f267f0bd3) > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072-config.diff, ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095361#comment-15095361 ] ASF subversion and git services commented on TS-3072: - Commit 230d25584fe358a3a8005766b92a5ce0a8681eac in trafficserver's branch refs/heads/6.1.x from shinrich [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=230d255 ] TS-3072: Debug logging for single connection. Without config control. This closes #398. (cherry picked from commit 45266942b4510f5d06f2af95ea7b8972134ad8f8) > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072-config.diff, ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095243#comment-15095243 ] ASF subversion and git services commented on TS-3072: - Commit 7cd050790b4cc6a85becebef7438e01f267f0bd3 in trafficserver's branch refs/heads/master from shinrich [ https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;h=7cd0507 ] TS-3072: fix newer compiler warnings. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072-config.diff, ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085640#comment-15085640 ] Leif Hedstrom commented on TS-3072: --- [~shinrich] Should we move this out to 6.2.0 ? > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072-config.diff, ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15085743#comment-15085743 ] Susan Hinrichs commented on TS-3072: We could merge in the changes in PR 398 without the config elements. Perhaps we will decide on an agreeable config mechanism by 6.2. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072-config.diff, ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15069810#comment-15069810 ] ASF GitHub Bot commented on TS-3072: Github user shinrich closed the pull request at: https://github.com/apache/trafficserver/pull/350 > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15069809#comment-15069809 ] ASF GitHub Bot commented on TS-3072: Github user shinrich commented on the pull request: https://github.com/apache/trafficserver/pull/350#issuecomment-166934284 Created new PR #398 > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15069814#comment-15069814 ] Susan Hinrichs commented on TS-3072: Must not have formatted the bug number correctly on the new PR. See https://github.com/apache/trafficserver/pull/398 for a PR without configuration logic. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15060891#comment-15060891 ] Alan M. Carroll commented on TS-3072: - This was discussed at the 6.1 bug scrub and the concensus was Susan should split the PR in to the base mechanism (which every is +1 about) and the enabling mechanism, which is the disputed issue. We'll commit the former for 6.1 and defer configuration / enabling until we have a better consensus. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036092#comment-15036092 ] ASF GitHub Bot commented on TS-3072: Github user SolidWallOfCode commented on the pull request: https://github.com/apache/trafficserver/pull/350#issuecomment-161358854 Uri - it is not the case that header_rewrite can support the required use case. There is no hook that is sufficiently early to enable pre-SSL accept debugging and therefore no plugin can actually enable it that early. I understand your point and in general agree with it, but the fact remains it is not possible to do it with the current set of hooks. If you want to propose adding a hook to make it possible (which would have to be immediately after the TCP handshake) that would be good. That is outside the scope of this PR. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036235#comment-15036235 ] ASF GitHub Bot commented on TS-3072: Github user sudheerv commented on the pull request: https://github.com/apache/trafficserver/pull/350#issuecomment-161377617 Uri - I agree with your point in general as well, but, I think it's arguable whether to treat this as a *feature* that isn't needed in the core. I can say, with certainty, that a majority (if not all) of production deployments would be benefitted from this capability. Having this in the core (even in its simplest form) already provides a great benefit to them, in terms of simplicity in configuration and use. Having to deploy a plugin and maintain it's configuration separately only makes it more complicated and less easy to use for general operations. Fwiw, we also have a number of similar features (which could have all been done using plugins too) in the core that are used extensively by operations and plugin developers alike (e.g @ headers). > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036183#comment-15036183 ] ASF GitHub Bot commented on TS-3072: Github user bryancall commented on the pull request: https://github.com/apache/trafficserver/pull/350#issuecomment-161370254 @ushachar I added support for debugging in header_rewrite and it has the ability to do client ip matching. As @SolidWallOfCode pointed out there may be some use cases where you might want to debug the TLS handshake and would require an earlier hook. I have felt this feature should be part of a plugin and it is in two already (header_rewrite and XDebug). If debugging required at an earlier stage add a hook and modify the plugins. If on the fly debugging is required then modify the existing plugins to reload their configurations. Added a third way to turn on debugging will only make things more confusing. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034059#comment-15034059 ] ASF GitHub Bot commented on TS-3072: Github user jpeach commented on the pull request: https://github.com/apache/trafficserver/pull/350#issuecomment-161030286 Yeh I mean the latter. All the code in ```ContFlags``` boils down to getspecific + setspecific, the rest is really not necessary. A separate file with a class that is used only once (and it not really re-usable) spreads the code around in a way that makes it harder to follow. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034077#comment-15034077 ] ASF GitHub Bot commented on TS-3072: Github user shinrich commented on the pull request: https://github.com/apache/trafficserver/pull/350#issuecomment-161034092 Originally, I just had an int pointer for the flags. Then we were doing experiments with multiple flags, and it seemed like having a separate class with bit names was more readable. From a runtime implementation perspective, I think it is all about the same. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034040#comment-15034040 ] ASF GitHub Bot commented on TS-3072: Github user SolidWallOfCode commented on the pull request: https://github.com/apache/trafficserver/pull/350#issuecomment-161026974 James - a `uintptr_t` instead of `ContFlags` would require the assumption a continuation always runs on the same thread, which is not the case. `ContFlags` ensures that the continuation gets the thread based flags regardless of the thread on which it executes. On the other hand if you mean doing the thread specific stuff in `Continuation` I would think it's clearly better to isolate that stuff in `ContFlags`. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035398#comment-15035398 ] ASF GitHub Bot commented on TS-3072: Github user ushachar commented on the pull request: https://github.com/apache/trafficserver/pull/350#issuecomment-161205132 To quote our own docs (under Admin Guide/Plugins): """ One of the key features of Apache Traffic Server™ is its modularity. Features that aren’t needed in the core simply aren’t there. This helps to provide an additional guarantee that our core can remain fast by concentrating on the things that we always provide: caching and proxying. """ I'd add that keeping things out of the core also makes it easier to understand, verify and debug. The way I see it - Logic on when to enable debugging on a transaction/session can be arbitrarily complex. I'm sure everyone on this thread can see the use of enabling debugs based on things like cipher suites, TLS protocol version, destination IPs. Even when relying only on source IPs, you might want to enable for only one in ten connections to cut the logging volume. Our (badly named :-) ) header_rewrite plugin can easily be adjusted to support such logic. (Personally - my trigger to enable debugging is often determined in an external process that does policy matching) @djcarlin - This being implemented in a plugin doesn't mean you'd have to restart ATS to use it -- The plugin would be loaded by default and you could enable/disable debugging by modifying it's configuration. (You could even talk to it directly via traffic_line using @SolidWallOfCode latest work on TS-4032) > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034021#comment-15034021 ] ASF GitHub Bot commented on TS-3072: Github user jpeach commented on the pull request: https://github.com/apache/trafficserver/pull/350#issuecomment-161024984 The whole ```ContFlags``` class seems unnecessary. If you really need per-thread flags, then why not just keep a thread-specify ```uintptr_t``` in the ```Continuation```? > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15032405#comment-15032405 ] ASF GitHub Bot commented on TS-3072: Github user djcarlin commented on the pull request: https://github.com/apache/trafficserver/pull/350#issuecomment-160751958 I like it being built into core - we can enable per-ip debugging without restarting ATS since the setting is overridable. This way we don't end up clearing the condition we were trying to debug. Its also a lot easier to use this way; I can enable via traffic_line setting versus configuring a plugin, draining traffic from the host, restarting ATS to load the plugin, etc. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15032429#comment-15032429 ] ASF GitHub Bot commented on TS-3072: Github user mlibbey commented on the pull request: https://github.com/apache/trafficserver/pull/350#issuecomment-160755930 We have health check and other "random" traffic to machines that make reading debug logs really painful. As a result, we are constantly wishing for a way to turn debugging on for a single remap rule. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026924#comment-15026924 ] ASF GitHub Bot commented on TS-3072: Github user sudheerv commented on the pull request: https://github.com/apache/trafficserver/pull/350#issuecomment-159640428 We can use the PR for voting? If an email is needed, perhaps, @shinrich might send a vote request, given that she did the current patch. In any case, my vote is +1 on having this feature in the core :) > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027355#comment-15027355 ] ASF GitHub Bot commented on TS-3072: Github user shinrich commented on the pull request: https://github.com/apache/trafficserver/pull/350#issuecomment-159698637 I'll send a note around next Monday when folks are back from Thanksgiving. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025679#comment-15025679 ] ASF GitHub Bot commented on TS-3072: Github user shinrich commented on the pull request: https://github.com/apache/trafficserver/pull/350#issuecomment-159436716 W.r.t. UnixNetVConnection pulling in Diags, that is kind of disturbing and can be cleaned up I think. The network area can read in the client_ip config and do the test. If we decide to keep the config-based control, I'll fix that up. Of course we are dragging in the global diag everywhere we are calling the Debug() macros. I removed the extra reference to Diags.h, and everything still built. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025639#comment-15025639 ] ASF GitHub Bot commented on TS-3072: Github user shinrich commented on the pull request: https://github.com/apache/trafficserver/pull/350#issuecomment-159433112 Ah, forgot to add the new file. Will get them in. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025484#comment-15025484 ] ASF GitHub Bot commented on TS-3072: Github user shinrich commented on the pull request: https://github.com/apache/trafficserver/pull/350#issuecomment-159413805 From my understanding of the code the SSN_START hooks happen after the SSL connections have been negotiated. So you would miss debug messages there. I would assume that you would miss some startup messages from HTTP2/SPDY too particularly since our SSN/TXN differentiation is mixed up after the addition of multiplexing protocols like HTTP2 and SPDY. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025504#comment-15025504 ] ASF GitHub Bot commented on TS-3072: Github user ushachar commented on the pull request: https://github.com/apache/trafficserver/pull/350#issuecomment-159416537 Well -- For SSL you could use the SNI callback, though I agree that adding a 'new connection' hookpoint would be nicer. To clarify -- I'm all for supporting the continuation based override (Note that it partially obsoletes the per HttpSM/ClientSession override that's already present). I'm just saying that I think the logic on when to turn it on should be in plugin space > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025505#comment-15025505 ] ASF GitHub Bot commented on TS-3072: Github user ushachar commented on the pull request: https://github.com/apache/trafficserver/pull/350#issuecomment-159416923 See the discussion here: http://wilderness.apache.org/channels/?f=traffic-server/2015-11-17#1447797537 > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[
https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025522#comment-15025522
]
ASF GitHub Bot commented on TS-3072:
Github user jpeach commented on the pull request:
https://github.com/apache/trafficserver/pull/350#issuecomment-159418105
```ContFlags.{cc,h}``` appear to be missing. I'm with @ushachar on this; I
think the capability is useful but I'm concerned about having the policy in the
core.
We need to be really careful about dependencies here. For example, I don't
know that ```P_UnixNetVConnection.h``` should drag in ```Diags.h```.
> Debug logging for a single connection in production traffic.
>
>
> Key: TS-3072
> URL: https://issues.apache.org/jira/browse/TS-3072
> Project: Traffic Server
> Issue Type: Improvement
> Components: Core, Logging
>Affects Versions: 5.0.1
>Reporter: Sudheer Vinukonda
>Assignee: Susan Hinrichs
> Labels: Yahoo
> Fix For: 6.1.0
>
> Attachments: ts-3072.diff
>
>
> Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is
> really hard to isolate/debug with the high traffic. Turning on debug logs in
> traffic is unfortunately not an option due to performance impacts. Even if
> you took a performance hit and turned on the logs, it is just as hard to
> separate out the logs for a single connection/transaction among the millions
> of the logs output in a short period of time.
> I think it would be good if there's a way to turn on debug logs in a
> controlled manner in production environment. One simple option is to support
> a config setting for example, with a client-ip, which when set, would turn on
> debug logs for any connection made by just that one client. If needed,
> instead of one client-ip, we may allow configuring up to 'n' (say, 5)
> client-ips.
> If there are other ideas, please comment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025488#comment-15025488 ] ASF GitHub Bot commented on TS-3072: Github user SolidWallOfCode commented on the pull request: https://github.com/apache/trafficserver/pull/350#issuecomment-159414518 Yes. You would need a new hook that gets called after the TCP connection (immediately after accept() returns the new socket). Also, how would the plugin enable debug messages for just that user agent connection? > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025414#comment-15025414 ] ASF GitHub Bot commented on TS-3072: GitHub user shinrich opened a pull request: https://github.com/apache/trafficserver/pull/350 TS-3072: Debug logging for single connection. Proposed changes to enable per-client-ip specification for debug messages. A version of this code has been running in our production for the past month. We are able to capture debug messages from the start of the TCP connection. You can merge this pull request into a Git repository by running: $ git pull https://github.com/shinrich/trafficserver ts-3072 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/trafficserver/pull/350.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #350 > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025469#comment-15025469 ] ASF GitHub Bot commented on TS-3072: Github user ushachar commented on the pull request: https://github.com/apache/trafficserver/pull/350#issuecomment-159412153 I had an IRC discussion with sudheerv about this change during the summit -- I strongly feel that if this can be implemented via a plugin (and I think it the SSN_START happens early enough), it should be. The bottom line of the IRC conversation was that he would bring this up for a vote. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950607#comment-14950607 ] Susan Hinrichs commented on TS-3072: Ignore my previous performance numbers. I had some servers missing when I ran that. I ran a cache scenario on my 1GB network machine and 4 multi-threaded clients. The clients fetch a cached 512 byte item. I'm still testing three cases. In addition to rps changes, I noted the change in CPU % utilization from perf top for Diags:on and pthread_get_specific. Base (not including this code change). 163125 rps. .97% in Diags::on and 0.61% in pthread_get_specific New (enable set to 0). 163169 rps. 1.19% in Diags:on and 0.58% in pthread_get_specific New (enable set to 2). 162777 rps. 1.3% Diags::on and 1.06% in pthread_get_specific So the impact of enabling the Debug IP checking but not actually matching (and logging) seems pretty minimal. In these experiments, we spend roughly extra 0.75% CPU and lose roughly 400 rps (or 0.2% reduction). No real impact in adding the code but leaving debug.enabled at 0. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: sometime > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950359#comment-14950359 ] Susan Hinrichs commented on TS-3072: [~zwoop] thanks for reminding me of duplexing. My rps had risen above 65K rps and I has very confused. [~bcall] agreed that for real performance testing we should be working on machines with 10G interfaces. For the purposes of this issue though I don't think we really care about absolute performance. We just need to have an understanding of the penalty of enabling a IP-specific debug, and verifying that this code change doesn't affect performance if debug is turned off entirely. I'll spend some more time today sorting out my test setup, and post updated comparisons and add numbers for the cached case as well. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: sometime > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950379#comment-14950379 ] Susan Hinrichs commented on TS-3072: [~jpe...@apache.org] let me ponder that some more. But here are my first thoughts. The transaction is not always (or easily) available from the VC level which is where many useful debug messages lie. We could push the debug_override flag from the continuation down into the NetworkVC class. As it turns out I only ended up using the debug_override on the netVC's. So that would eliminate polluting the top Continuation class. It would be nice to not use thread local storage. The motivation to use thread local storage was to minimize code change, and ease the inclusion of future Debug messages into the conditional debug scheme. I'll look again to see of other data structures are always available to Diag and the point of debug decision making. One could add a Plugin call to adjust the debug_override flag from the transaction object (assuming one could get access to the netvc from the transaction) or from the session object. Though I guess a tricky bit of doing a per transaction debug they way I have things set up is only debugging one transaction but not the others on the same net VC (in the case of HTTP2 or SPDY). > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: sometime > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951033#comment-14951033 ] Leif Hedstrom commented on TS-3072: --- Nice! 400 RPS is within "noise", i see that sort of Jitter in all my benchmarks. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: sometime > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951043#comment-14951043 ] Bryan Call commented on TS-3072: As long as you are not maxing out the NIC it should be good. If you are maxing out the NIC then there will be other issues that will change the results of the benchmark. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: 6.1.0 > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14948975#comment-14948975 ] Susan Hinrichs commented on TS-3072: I ran some tests on my test harness machines. It is configured in proxy mode (no caching) making GET requests. 1KB objects are exchanged. Three requests per connection. No SSL. Single 1Gb interface. I ran my clients at not quite resource exhaustion to measure a steady state performance. Run cases: * Base: A build without client_ip debug code. diags.debug enabled = 0 * New0: A build with client_ip debug code. diags.debug enabled = 0 * New2-mismatch: A build with client_ip debug code. diags.debug.enabled = 2 and diags.debug.client_ip = IP address not involved in the test. I also tried running with an IP matching one of my test clients and http tag set, the client fell over. Base and New0 had very similar performance. About 56,600 rps. New2-mismatch was around 53,900 rps. So enabling the client_ip even if nothing matches incurs around a 5% performance penalty. But this is only seen if the configuration is explicitly set from 0 to 2. In the depths of an investigation, this might be an acceptable performance penalty. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: sometime > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949072#comment-14949072 ] Bryan Call commented on TS-3072: The rps seems low. I have been able to do 250K rps before. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: sometime > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949174#comment-14949174 ] Susan Hinrichs commented on TS-3072: Poking at my test some more and the numbers are a bit higher. But based on my math, the single 1Gbps connection puts a hard limit of 65536 rps when running without caching and having each request exchange 1KB. Max bytes sent in a second = 1024*1024*1024/8 > number of bytes sent in T transactions = 1024*T*2 1024*1024/16 > T 65536 > T > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: sometime > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949418#comment-14949418 ] Leif Hedstrom commented on TS-3072: --- 1G links should be able to do over 100MB / sec, so even at 1KB objects, you should be able to see over 100K QPS. I usually test with 100 byte objects, which including the header is less than 512 bytes, and it can therefore do roughly 150-200k QPS depending on hardware. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: sometime > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949587#comment-14949587 ] Bryan Call commented on TS-3072: We should be using 10 gige nics for benchmarking. Most of the major contributors to the project are running 10 gige in production. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda >Assignee: Susan Hinrichs > Labels: Yahoo > Fix For: sometime > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[
https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949860#comment-14949860
]
James Peach commented on TS-3072:
-
I think this is a great idea and really useful. I don't much like how the
implementation depends on changes to {{Continuation}} and thread locals. Is is
possible to implement this by tagging the transaction itself?
> Debug logging for a single connection in production traffic.
>
>
> Key: TS-3072
> URL: https://issues.apache.org/jira/browse/TS-3072
> Project: Traffic Server
> Issue Type: Improvement
> Components: Core, Logging
>Affects Versions: 5.0.1
>Reporter: Sudheer Vinukonda
>Assignee: Susan Hinrichs
> Labels: Yahoo
> Fix For: sometime
>
> Attachments: ts-3072.diff
>
>
> Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is
> really hard to isolate/debug with the high traffic. Turning on debug logs in
> traffic is unfortunately not an option due to performance impacts. Even if
> you took a performance hit and turned on the logs, it is just as hard to
> separate out the logs for a single connection/transaction among the millions
> of the logs output in a short period of time.
> I think it would be good if there's a way to turn on debug logs in a
> controlled manner in production environment. One simple option is to support
> a config setting for example, with a client-ip, which when set, would turn on
> debug logs for any connection made by just that one client. If needed,
> instead of one client-ip, we may allow configuring up to 'n' (say, 5)
> client-ips.
> If there are other ideas, please comment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14945245#comment-14945245 ] Susan Hinrichs commented on TS-3072: Only ad hoc performance comparison so far. I'll run a sequence of tests on the stress test box. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda > Labels: Yahoo > Fix For: sometime > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944479#comment-14944479 ] Leif Hedstrom commented on TS-3072: --- Seems reasonable. Was this benchmarked? A test which is running at 100k or more QPS would be useful to verify that there not a performance issue, at least when not enabled. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda > Labels: Yahoo > Fix For: sometime > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944464#comment-14944464 ] Sudheer Vinukonda commented on TS-3072: --- Needless to say, I'm +1 on this feature being in the core with the ability to control with a simple config settings. The patch looks cool and simple and I love the fact that it will automatically output any new logs without having to make any additional changes. > Debug logging for a single connection in production traffic. > > > Key: TS-3072 > URL: https://issues.apache.org/jira/browse/TS-3072 > Project: Traffic Server > Issue Type: Improvement > Components: Core, Logging >Affects Versions: 5.0.1 >Reporter: Sudheer Vinukonda > Labels: Yahoo > Fix For: sometime > > Attachments: ts-3072.diff > > > Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is > really hard to isolate/debug with the high traffic. Turning on debug logs in > traffic is unfortunately not an option due to performance impacts. Even if > you took a performance hit and turned on the logs, it is just as hard to > separate out the logs for a single connection/transaction among the millions > of the logs output in a short period of time. > I think it would be good if there's a way to turn on debug logs in a > controlled manner in production environment. One simple option is to support > a config setting for example, with a client-ip, which when set, would turn on > debug logs for any connection made by just that one client. If needed, > instead of one client-ip, we may allow configuring up to 'n' (say, 5) > client-ips. > If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[
https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14333597#comment-14333597
]
James Peach commented on TS-3072:
-
I'll be committing {{traffic_ctl}} today. From now on, please add any
management support to {{traffic_ctl}} rather than {{traffic_line}}.
Debug logging for a single connection in production traffic.
Key: TS-3072
URL: https://issues.apache.org/jira/browse/TS-3072
Project: Traffic Server
Issue Type: Improvement
Components: Core, Logging
Affects Versions: 5.0.1
Reporter: Sudheer Vinukonda
Labels: Yahoo
Fix For: 5.3.0
Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is
really hard to isolate/debug with the high traffic. Turning on debug logs in
traffic is unfortunately not an option due to performance impacts. Even if
you took a performance hit and turned on the logs, it is just as hard to
separate out the logs for a single connection/transaction among the millions
of the logs output in a short period of time.
I think it would be good if there's a way to turn on debug logs in a
controlled manner in production environment. One simple option is to support
a config setting for example, with a client-ip, which when set, would turn on
debug logs for any connection made by just that one client. If needed,
instead of one client-ip, we may allow configuring up to 'n' (say, 5)
client-ips.
If there are other ideas, please comment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[
https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14137610#comment-14137610
]
James Peach commented on TS-3072:
-
I would guess that {{TSHttpTxnDebugGet/Set}} mostly addresses this need.
Debug logging for a single connection in production traffic.
Key: TS-3072
URL: https://issues.apache.org/jira/browse/TS-3072
Project: Traffic Server
Issue Type: Improvement
Components: Core, Logging
Affects Versions: 5.0.1
Reporter: Sudheer Vinukonda
Labels: Yahoo
Fix For: 5.2.0
Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is
really hard to isolate/debug with the high traffic. Turning on debug logs in
traffic is unfortunately not an option due to performance impacts. Even if
you took a performance hit and turned on the logs, it is just as hard to
separate out the logs for a single connection/transaction among the millions
of the logs output in a short period of time.
I think it would be good if there's a way to turn on debug logs in a
controlled manner in production environment. One simple option is to support
a config setting for example, with a client-ip, which when set, would turn on
debug logs for any connection made by just that one client. If needed,
instead of one client-ip, we may allow configuring up to 'n' (say, 5)
client-ips.
If there are other ideas, please comment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TS-3072) Debug logging for a single connection in production traffic.
[ https://issues.apache.org/jira/browse/TS-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14137626#comment-14137626 ] Sudheer Vinukonda commented on TS-3072: --- Thanks, [~jpeach] - Yes, I am using the same framework as that of TSHttpTxnDebugGet()/Set(), except that I'm making it a little bit more easier to use (for e.g. adding a setting for the user to specify any arbitrary client/ip using traffic_line rather than having to write/code a plugin etc) and also extend the logging support beyond http (for e.g. NetVC, Spdy, FetchSM etc). Currently, we have it only in HttpSM/HttpTransact. Debug logging for a single connection in production traffic. Key: TS-3072 URL: https://issues.apache.org/jira/browse/TS-3072 Project: Traffic Server Issue Type: Improvement Components: Core, Logging Affects Versions: 5.0.1 Reporter: Sudheer Vinukonda Labels: Yahoo Fix For: 5.2.0 Presently, when there's a production issue (e.g. TS-3049, TS-2983 etc), it is really hard to isolate/debug with the high traffic. Turning on debug logs in traffic is unfortunately not an option due to performance impacts. Even if you took a performance hit and turned on the logs, it is just as hard to separate out the logs for a single connection/transaction among the millions of the logs output in a short period of time. I think it would be good if there's a way to turn on debug logs in a controlled manner in production environment. One simple option is to support a config setting for example, with a client-ip, which when set, would turn on debug logs for any connection made by just that one client. If needed, instead of one client-ip, we may allow configuring up to 'n' (say, 5) client-ips. If there are other ideas, please comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
