[jira] Updated: (TS-489) Seg Fault with Connection_Collapsing and clustering enabled.
[ https://issues.apache.org/jira/browse/TS-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mohan_zl updated TS-489: Attachment: ts_489_testing.txt We did an experiment with ts2.0.1 stable version following the way you setup the cluster and changed the relevant arguments, and the results demonstrate that we can use the full cluster mode and connection_collapsing feature together in ts2.0.1 version. The attach file shows how each change affects the results. Seg Fault with Connection_Collapsing and clustering enabled. Key: TS-489 URL: https://issues.apache.org/jira/browse/TS-489 Project: Traffic Server Issue Type: Bug Affects Versions: 2.0.0 Environment: Debian Lenny. 2.6.26-2-amd-64 Sun Blade X6240 (2 x Six-Core AMD Opteron(tm) Processor 2439 SE) 64G Memory Reporter: Ricky Chan Fix For: 2.3.0 Attachments: collapse1.trace, collapse2.trace, ts_489_testing.txt Bug is easily reproduced, with the following setup. Traffic Server 2.0.0 Enable Clustering (so you'll need two machine and make sure cluster is actually working) (LOCAL proxy.local.cluster.type INT 1) Enable Connection Collapsing (CONFIG proxy.config.connection_collapsing.hashtable_enabled INT 1) Other changes to records.config which may or may affect it are changes to heuristics: CONFIG proxy.config.http.cache.heuristic_min_lifetime INT 5 CONFIG proxy.config.http.cache.heuristic_max_lifetime INT 86400 CONFIG proxy.config.http.cache.heuristic_lm_factor FLOAT 0.000100 CONFIG proxy.config.http.cache.fuzz.time INT 240 CONFIG proxy.config.http.cache.fuzz.probability FLOAT 0.05 Using a 3rd machine using apache benchmark (ab) and request with say -n 100 with keep alive (-k) and -c 8000 say. I found it happens all the time above 8000. I just fetched a file from origin on lighttpd which had a cache-control header of max-age 86400, so to reduce hitting origin. Size of file is 9 bytes only. Note: You need to set ulimit -n very high and set sysctl ip_local_port_range to larger than defaults to be able to run test, I did ulimit -n 100 and had sysctl -w net.ipv4.ip_local_port_range=1024 65000 to be able to run AB. Disabling clustering or connection Collapsing the program no longer. I then added GDB wrapper around traffic_server and it clearly shows it's the connection collapsing API which is at fault here. I'll add these traces as attachments. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (TS-489) Seg Fault with Connection_Collapsing and clustering enabled.
[ https://issues.apache.org/jira/browse/TS-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12925351#action_12925351 ] Ricky Chan commented on TS-489: --- I'll retest with 2.0.1, although I did do a differ between 2.0.0 and 2.0.1 when I first encounter this issue and didn't see any IMO code changes which was related to this. I just saw mianly the 3 fixes mentioned in the release fix. Personally I'll repeat the test with -c 20,000 and have it run for a long period. Also keep an eye on the origin, as a tell tale sign is a sudden rush of 1000;s of direct connections to the origin. I'll retest with 2.0.1 with my setup and I'll get back to you. Ricky p.s. Zhao, I'm happy to later versions when I get a chance to. Seg Fault with Connection_Collapsing and clustering enabled. Key: TS-489 URL: https://issues.apache.org/jira/browse/TS-489 Project: Traffic Server Issue Type: Bug Affects Versions: 2.0.0 Environment: Debian Lenny. 2.6.26-2-amd-64 Sun Blade X6240 (2 x Six-Core AMD Opteron(tm) Processor 2439 SE) 64G Memory Reporter: Ricky Chan Fix For: 2.3.0 Attachments: collapse1.trace, collapse2.trace, ts_489_testing.txt Bug is easily reproduced, with the following setup. Traffic Server 2.0.0 Enable Clustering (so you'll need two machine and make sure cluster is actually working) (LOCAL proxy.local.cluster.type INT 1) Enable Connection Collapsing (CONFIG proxy.config.connection_collapsing.hashtable_enabled INT 1) Other changes to records.config which may or may affect it are changes to heuristics: CONFIG proxy.config.http.cache.heuristic_min_lifetime INT 5 CONFIG proxy.config.http.cache.heuristic_max_lifetime INT 86400 CONFIG proxy.config.http.cache.heuristic_lm_factor FLOAT 0.000100 CONFIG proxy.config.http.cache.fuzz.time INT 240 CONFIG proxy.config.http.cache.fuzz.probability FLOAT 0.05 Using a 3rd machine using apache benchmark (ab) and request with say -n 100 with keep alive (-k) and -c 8000 say. I found it happens all the time above 8000. I just fetched a file from origin on lighttpd which had a cache-control header of max-age 86400, so to reduce hitting origin. Size of file is 9 bytes only. Note: You need to set ulimit -n very high and set sysctl ip_local_port_range to larger than defaults to be able to run test, I did ulimit -n 100 and had sysctl -w net.ipv4.ip_local_port_range=1024 65000 to be able to run AB. Disabling clustering or connection Collapsing the program no longer. I then added GDB wrapper around traffic_server and it clearly shows it's the connection collapsing API which is at fault here. I'll add these traces as attachments. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (TS-489) Seg Fault with Connection_Collapsing and clustering enabled.
[ https://issues.apache.org/jira/browse/TS-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12925360#action_12925360 ] Zhao Yongming commented on TS-489: -- cool Mohan and me have setup up a testing according your direction on v2.0.1, but we don't find the problem you got. btw, the cluster function may not works due to TS-390 in trunk. Seg Fault with Connection_Collapsing and clustering enabled. Key: TS-489 URL: https://issues.apache.org/jira/browse/TS-489 Project: Traffic Server Issue Type: Bug Affects Versions: 2.0.0 Environment: Debian Lenny. 2.6.26-2-amd-64 Sun Blade X6240 (2 x Six-Core AMD Opteron(tm) Processor 2439 SE) 64G Memory Reporter: Ricky Chan Fix For: 2.3.0 Attachments: collapse1.trace, collapse2.trace, ts_489_testing.txt Bug is easily reproduced, with the following setup. Traffic Server 2.0.0 Enable Clustering (so you'll need two machine and make sure cluster is actually working) (LOCAL proxy.local.cluster.type INT 1) Enable Connection Collapsing (CONFIG proxy.config.connection_collapsing.hashtable_enabled INT 1) Other changes to records.config which may or may affect it are changes to heuristics: CONFIG proxy.config.http.cache.heuristic_min_lifetime INT 5 CONFIG proxy.config.http.cache.heuristic_max_lifetime INT 86400 CONFIG proxy.config.http.cache.heuristic_lm_factor FLOAT 0.000100 CONFIG proxy.config.http.cache.fuzz.time INT 240 CONFIG proxy.config.http.cache.fuzz.probability FLOAT 0.05 Using a 3rd machine using apache benchmark (ab) and request with say -n 100 with keep alive (-k) and -c 8000 say. I found it happens all the time above 8000. I just fetched a file from origin on lighttpd which had a cache-control header of max-age 86400, so to reduce hitting origin. Size of file is 9 bytes only. Note: You need to set ulimit -n very high and set sysctl ip_local_port_range to larger than defaults to be able to run test, I did ulimit -n 100 and had sysctl -w net.ipv4.ip_local_port_range=1024 65000 to be able to run AB. Disabling clustering or connection Collapsing the program no longer. I then added GDB wrapper around traffic_server and it clearly shows it's the connection collapsing API which is at fault here. I'll add these traces as attachments. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (TS-498) New configuration for dedicated DNS thread
New configuration for dedicated DNS thread -- Key: TS-498 URL: https://issues.apache.org/jira/browse/TS-498 Project: Traffic Server Issue Type: Improvement Components: Documentation Reporter: Leif Hedstrom Priority: Minor Fix For: 2.2.0 *) Experimental supoprt for a dedicated DNS thread. This can be enabled with the records.config option CONFIG proxy.config.dns.dedicated_thread INT 1 This feature is possibly useful for very busy forward or transparent proxies [TS-307]. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (TS-489) Seg Fault with Connection_Collapsing and clustering enabled.
[ https://issues.apache.org/jira/browse/TS-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12925381#action_12925381 ] Ricky Chan commented on TS-489: --- Okay I did a quick re-run of my tests user the 2.0.1 stable release. I ran this for about 30 minutes, and with -c 20,000. Here is my results: Firstly it no longer crashes and AB will show successful results, However looking at the origin logs, I can see that cache data is not working as expected. Firstly I get 1,000's of queries to the origins, i.e. collapsing caching is no longer obeyed. Secondly, every few seconds it will make more queries even though cache for max-age is set to 86400 and heuristics is set to a rate where it shouldn't come back that quickly. I then examined TS logs, and I can see a load of TCP_MISS's. I then repeat the test with clustering disabled. No hits to the origin at all (as it has a cached version already), I then purge the content. from the cache, and I see 1 connection to the origin as expect even though I am requesting 20,000 copies of it (so collapsing is working as expected). So these 2 features in 2.0.1 (albeit without crashing) in my tests are not working properly, hitting the origin in those volumes are not possible acceptable for my downstream origin owners. Mohan, check you logs (origin and TS) and make sure they have behaved as expected, because I suspect not. Seg Fault with Connection_Collapsing and clustering enabled. Key: TS-489 URL: https://issues.apache.org/jira/browse/TS-489 Project: Traffic Server Issue Type: Bug Affects Versions: 2.0.0 Environment: Debian Lenny. 2.6.26-2-amd-64 Sun Blade X6240 (2 x Six-Core AMD Opteron(tm) Processor 2439 SE) 64G Memory Reporter: Ricky Chan Fix For: 2.3.0 Attachments: collapse1.trace, collapse2.trace, ts_489_testing.txt Bug is easily reproduced, with the following setup. Traffic Server 2.0.0 Enable Clustering (so you'll need two machine and make sure cluster is actually working) (LOCAL proxy.local.cluster.type INT 1) Enable Connection Collapsing (CONFIG proxy.config.connection_collapsing.hashtable_enabled INT 1) Other changes to records.config which may or may affect it are changes to heuristics: CONFIG proxy.config.http.cache.heuristic_min_lifetime INT 5 CONFIG proxy.config.http.cache.heuristic_max_lifetime INT 86400 CONFIG proxy.config.http.cache.heuristic_lm_factor FLOAT 0.000100 CONFIG proxy.config.http.cache.fuzz.time INT 240 CONFIG proxy.config.http.cache.fuzz.probability FLOAT 0.05 Using a 3rd machine using apache benchmark (ab) and request with say -n 100 with keep alive (-k) and -c 8000 say. I found it happens all the time above 8000. I just fetched a file from origin on lighttpd which had a cache-control header of max-age 86400, so to reduce hitting origin. Size of file is 9 bytes only. Note: You need to set ulimit -n very high and set sysctl ip_local_port_range to larger than defaults to be able to run test, I did ulimit -n 100 and had sysctl -w net.ipv4.ip_local_port_range=1024 65000 to be able to run AB. Disabling clustering or connection Collapsing the program no longer. I then added GDB wrapper around traffic_server and it clearly shows it's the connection collapsing API which is at fault here. I'll add these traces as attachments. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
