[squid-users] Re: Automatic StoreID ?
This is how Rock store does it, essentially: Rock store index does not store the real location of the object on disk but computes it based on the hash value. Sorry, then I misunderstood something, when reading some rock-code while ago. For me, in essence, it looked like, that for caching an object, rock picks one (or multiple, for large-rock) of the available slots for storage, and keeps the mapping hash-slot in the memory table. So, on restart, squid has to scan all slots from disk, to rebuild the table. Which means, the mapping URL-hash - slot_# is _not_ fixed (predictable). Positive consequence: No rebuild of the in-memory-table necessary, as there is none. Avoids the time-comsuning rebuild of rock-storage-table from disk. If you do not build the index, you have to do a disk I/O to fetch the first slot of the candidate object on _every_ request. Not necessarily to do a disk I/O, but to do an I/O. Still, underlying OS-buffering/blocking is happening. Besides, for a HIT you have to do the I/O anyway. So, the amount of unnecessary disk-I/Os would be the (squid-MISSes - not in OS/buffers residing disk-blocks). Which leads to a good compromise: Direct hashing would allow the slow population of the optional translation-table. -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Automatic-StoreID-tp4665140p4665204.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] Is it possible to mark tcp_outgoing_mark (server side) with SAME MARK as incoming packet (client side)?
On 15/03/2014 6:46 p.m., Amm wrote: Hello, I would like to mark outgoing packet (on server side) with SAME MARK as on incoming (NATed or CONNECTed) packet. There is option tcp_outgoing_mark with which I can mark packets. But there is no ACL option to check incoming mark. If there is already a way to do this then please guide. No patch is needed to preserve the netfilter mark, which will work with all variants of netfilter. http://www.squid-cache.org/Doc/config/qos_flows/ Squid default action is to pass the netfilter MARK value from client through to the server. All you should need to do is *omit* tcp_outgoing_mark directives from changing it to something else. Amos
[squid-users] Re: Some problem about use squid
Amos: I am try to set the log to ms-sql always ,but have not succeed . when we set: logfile_daemon /path/log_db_daemon or /access_log daemon:/server/squid/access_log/squid/squid_password and exec #./squid -k reconfigure or #./squid -k shutdown it say: squid: ERROR: Could not send signal 1 to process /5662/: (3) No such process -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Some-problem-about-use-squid-tp4663845p4665206.html Sent from the Squid - Users mailing list archive at Nabble.com.
[squid-users] couldn't cache this image file
Can anybody get this image to be cached? If yes, in which squid version? http://s2.glbimg.com/XG7L47pX_ik0O_uxiV3g65rvpNU=/90x68/s.glbimg.com/jo/g1/f/original/2014/02/21/whatsapp.jpg I tried with squid 3.3.12, which came with this bug correction: (Bug #3806: Caching responses with Vary header) Tried with 3.3.8 too, but with no luck. Only MISS responses. thanks, Carlos
Re: [squid-users] Is it possible to mark tcp_outgoing_mark (server side) with SAME MARK as incoming packet (client side)?
On 03/15/2014 05:11 PM, Amos Jeffries wrote: On 15/03/2014 6:46 p.m., Amm wrote: I would like to mark outgoing packet (on server side) with SAME MARK as on incoming (NATed or CONNECTed) packet. http://www.squid-cache.org/Doc/config/qos_flows/ Squid default action is to pass the netfilter MARK value from client through to the server. All you should need to do is *omit* tcp_outgoing_mark directives from changing it to something else. Amos Oh that's great, thanks, I did not know this. However, I tried this but somehow I am not able to get it working Please let me know what could be wrong. First I thought it may be because netfilter-conntrack-devel was not installed. So I installed the same. Then I recompiled squid with these: --with-netfilter-conntrack and --with-libcap configure: ZPH QOS enabled: yes configure: QOS netfilter mark preservation enabled: yes ... checking for operational libcap2 headers... yes configure: libcap support enabled: yes configure: libcap2 headers are ok: yes ... configure: Linux Netfilter support requested: yes configure: Linux Netfilter Conntrack support requested: yes checking for library containing nfct_query... -lnetfilter_conntrack (4-5 more lines with header check with answer yes) Installed new squid and restarted squid. Ran following iptables command for debugging: # CMD 1- mark all packets coming from 192.168.1.45 $ iptables -t mangle -I PREROUTING -s 192.168.1.45 -j MARK --set-mark 0x112 # CMD 2 - count packets/bytes going OUT on port 80 and marked 0x112 $ iptables -t mangle -I POSTROUTING -m mark --mark 0x112 -p tcp --dport 80 # CMD 3 - NAT settings (intercept) $ iptables -t nat -nvL Chain PREROUTING (policy ACCEPT 22610 packets, 2251K bytes) pkts bytes target prot opt in out source destination 347 21371 REDIRECT tcp -- eth0 * 0.0.0.0/0 0.0.0.0/0tcp dpt:80 redir ports 3128 Some settings in /etc/squid/squid.conf: http_port 3128 intercept # log for nfmark logging logformat nfmark %ts.%03tu %6tr %a %Ss/%03Hs %st %rm %ru %[un %Sh/%a %mt %nfmark %nfmark access_log daemon:/var/log/squid/access.log squid all access_log daemon:/var/log/squid/nfmark.log nfmark all (Do I need to put anything else in squid.conf for marking?) (There is no tcp_outgoing_mark) Now I accessed Google from 192.168.1.45 $ tail /var/log/squid/nfmark.log 1394891128.585403 192.168.1.45 TCP_MISS/200 21137 GET http://www.google.co.in/?xxx - HIER_DIRECT/173.194.36.56 text/html 0x0 0x0 1394891128.793 92 192.168.1.45 TCP_MISS/304 393 GET http://www.google.co.in/images/srpr/mlogo2x_3.png - HIER_DIRECT/173.194.36.56 - 0x0 0x0 1394891128.851115 192.168.1.45 TCP_MISS/304 393 GET http://www.google.co.in/images/logo_mobile_srp_3.png - HIER_DIRECT/173.194.36.56 - 0x0 0x0 nfmark in and out both are logged as 0x0 whereas I was expecting atleast one of them to be 0x112 $ iptables -t mangle -nvL PREROUTING Chain PREROUTING (policy ACCEPT 1590 packets, 604K bytes) pkts bytes target prot opt in out source destination 135 22042 MARK all -- * * 192.168.1.45 0.0.0.0/0MARK set 0x112 $ iptables -t mangle -nvL POSTROUTING Chain POSTROUTING (policy ACCEPT 1653 packets, 372K bytes) pkts bytes target prot opt in out source destination 0 0tcp -- * * 0.0.0.0/0 0.0.0.0/0mark match 0x112 multiport dports 80,443 PREROUTING shows 135 packets MARKed as 0x112 but POSTROUTING shows no packets marked. What could be wrong? Thanks in advance. Amm
[squid-users] Cygwin SSL Bumping
Is it possible to have SSL bumping with squid on Cygwin? I think to have SSL bumping, squid for cygwin needs to be recompiled with some additional options enabled. Is there an easy way to do this? I've been banging my head against the wall trying to get this to work.. Thanks in advance! Cheers, Derel
Re: [squid-users] Is it possible to mark tcp_outgoing_mark (server side) with SAME MARK as incoming packet (client side)?
On 03/15/2014 08:03 PM, Amm wrote: On 03/15/2014 05:11 PM, Amos Jeffries wrote: On 15/03/2014 6:46 p.m., Amm wrote: I would like to mark outgoing packet (on server side) with SAME MARK as on incoming (NATed or CONNECTed) packet. http://www.squid-cache.org/Doc/config/qos_flows/ Squid default action is to pass the netfilter MARK value from client through to the server. All you should need to do is *omit* tcp_outgoing_mark directives from changing it to something else. Amos Oh that's great, thanks, I did not know this. However, I tried this but somehow I am not able to get it working Please let me know what could be wrong. Ok I read further on that link itself, somewhere it says: disable-preserve-miss This option disables the preservation of the TOS or netfilter mark. By default, the existing TOS or netfilter mark value of the response coming from the remote server will be retained and masked with miss-mark. NOTE: in the case of a netfilter mark, the mark must be set on the connection (using the CONNMARK target) not on the packet (MARK target). First, it says to use CONNMARK and not MARK. I tried with CONNMARK as well but it did not work. Second, it says its for response coming from the remote server. My question however was to pass on mark from client side to server side. i.e. reverse of what above paragraph says. (But your earlier reply said client to server - so there is confusion) Any idea? Regards Amm
[squid-users] How to log IP address after successful authentication
Hi all, I need save IP address after successful authentication to log file, but I need only one record per authenticated user. I have username and password save in file therefore I would like to use NCSA_auth but I dont know how to get IP address in plugin. Second solution that I tried to use are external ACLs but there are no way how to get password to my auth plugin. Does anyone have any advice on how to save IP address after successful authentication??? Pavel
[squid-users] How to log IP address after successful authentication
Hi all, I need save IP address after successful authentication to log file, but I need only one record per authenticated user. I have username and password save in file therefore I would like to use NCSA_auth but I dont know how to get IP address in plugin. Second solution that I tried to use are external ACLs but there are no way how to get password to my auth plugin. Does anyone have any advice on how to save IP address after successful authentication??? Pavel
Re: [squid-users] Is it possible to mark tcp_outgoing_mark (server side) with SAME MARK as incoming packet (client side)?
On Sat, 2014-03-15 at 21:13 +0530, Amm wrote: Ok I read further on that link itself, somewhere it says: disable-preserve-miss This option disables the preservation of the TOS or netfilter mark. By default, the existing TOS or netfilter mark value of the response coming from the remote server will be retained and masked with miss-mark. NOTE: in the case of a netfilter mark, the mark must be set on the connection (using the CONNMARK target) not on the packet (MARK target). First, it says to use CONNMARK and not MARK. I tried with CONNMARK as well but it did not work. Yes, you definitely need to use CONNMARK, not MARK. Second, it says its for response coming from the remote server. I used (and created) the patch to get the value from the remote server. However, I can't remember whether it does it the other way as well (at the time I thought I'd written the documentation so clearly, but coming back to it now it's not clear...) From memory, however, you do need to configure qos_flows to *something*, to trigger its operation. I think you can simply state qos_flows mark. My question however was to pass on mark from client side to server side. i.e. reverse of what above paragraph says. As above, it's primarily server to client. Get that working first so you know everything is in order, and then try it the other way. Let me know what you find out and I will update the documentation! (I don't have time to look through the source code right now) Andy
Re: [squid-users] How to log IP address after successful authentication
On 16/03/2014 7:00 a.m., divispa wrote: Hi all, I need save IP address after successful authentication to log file, but I need only one record per authenticated user. I have username and password save in file therefore I would like to use NCSA_auth but I dont know how to get IP address in plugin. Second solution that I tried to use are external ACLs but there are no way how to get password to my auth plugin. Does anyone have any advice on how to save IP address after successful authentication??? There are solutions to each of yrou questions. But before I write them can you please explain a bit what you are trying to do with this IP address? I get the impression you are trying to do something with security based on the wrong assumption that each user has only 1 IP address. Amos
Re: [squid-users] Re: Some problem about use squid
On 16/03/2014 1:25 a.m., huang.zhenxing wrote: Amos: I am try to set the log to ms-sql always ,but have not succeed . when we set: logfile_daemon /path/log_db_daemon or /access_log daemon:/server/squid/access_log/squid/squid_password What is your *exact* squid.conf settings for those? You may mail me directly if they contain login to the DB server. and exec #./squid -k reconfigure or #./squid -k shutdown it say: squid: ERROR: Could not send signal 1 to process /5662/: (3) No such process Try squid -k parse to verify the configuration before reconfiguring. That way you will avoid causing an outage if there is any syntax errors in the file. Now that you have this error, check that the squid PID file matches the process ID of the running Squid (or that the file does not exist if Squid is not running). Amos
Re: [squid-users] logrotate only instead (all) squid rotate
On 15/03/2014 12:26 a.m., Alfredo Rezinovsky wrote: Using: squid -k rotate squid rotates logs but also closes and reopen caches_dirs and url_rewrite_programs There's a way to signal only the (logfile-daemon) processes to rotate the logs and only the logs ? Not with any current Squid. The helpers error reporting (stderr) channels are plugged directly into cache.log. So they need to be detached and re-attached by restarting when that log is moved. The cache journal(s) need cleaning up occasionally and are done as well on rotation. If you are using a daemon there should be no need to rotate much. The daemon is fully responsible for the access.logs and can rotate them even without an explicit command from Squid. For example; the log_file_daemon bundled with the latest Squid will rotate access.log on 32-bit systems whenever the current file exceeds 32-bit file sizes and starts getting errors. On the other hand the daemon helper is a separate helper partially so you can write your own and have it do whatever you like. If you do extend the bundled file daemon please consider submitting the patch for communal benefit. Also, why exactly are you asking for this? perhapse it does not exist yet because what you are needing is better or easier done some other way. Amos
Re: [squid-users] Delay Pools
On 14/03/2014 4:34 p.m., csn233 wrote:
Testing delay_pools on 3.3.11 (and other versions too)
delay_pools 1
delay_class 1 3
delay_parameters 1 -1/-1 -1/-1 128000/256000
delay_access 1 allow all
This works as expected. However, when requests goes a bit higher, say
20 req/s, the CPU shoots up to 100%, and strace shows epoll calls:
epoll_ctl(6, EPOLL_CTL_MOD, 7251, {EPOLLIN|EPOLLOUT|EPOLLERR|EPOLLHUP,
{u32=7251, u64=15832364667084217427}}) = 0
epoll_ctl(6, EPOLL_CTL_MOD, 8068, {EPOLLIN|EPOLLOUT|EPOLLERR|EPOLLHUP,
{u32=8068, u64=14612651267063816068}}) = 0
epoll_wait(6, {{EPOLLOUT, {u32=46, u64=3639753988997382190}},
{EPOLLOUT, {u32=7111, u64=16029163435841297351}}, {EPOLLOUT,.. [snip]
Is this normal?
Maybe yes, maybe no.
It is normal to see that type of fast looping when there is a lot of
incoming connections or TCP buffers constantly providing new bytes for
handling.
It may be expected when the delay pool slows down reading out of TCP
buffers and causes traffic to become backlogged.
However, I think sockets hitting the delay limits should be omitted from
the polling until they had more bytes allowed. So this may be a bug in
the mechanism doing that.
Next, I also tried client_delay_pools (3.3.11/3.4.3)
client_delay_pools 1
client_delay_access 1 allow all
client_delay_parameters 1 128000 256000
This gets connection reset straightaway. What am I missing?
Information about what the connection reset is coming from?
Is squid crashing? http://bugs.squid-cache.org/show_bug.cgi?id=3696
Amos
Re: [squid-users] logrotate only instead (all) squid rotate
El 15/03/14 22:21, Amos Jeffries escribió: On 15/03/2014 12:26 a.m., Alfredo Rezinovsky wrote: Using: squid -k rotate squid rotates logs but also closes and reopen caches_dirs and url_rewrite_programs There's a way to signal only the (logfile-daemon) processes to rotate the logs and only the logs ? Not with any current Squid. The helpers error reporting (stderr) channels are plugged directly into cache.log. So they need to be detached and re-attached by restarting when that log is moved. The cache journal(s) need cleaning up occasionally and are done as well on rotation. If you are using a daemon there should be no need to rotate much. The daemon is fully responsible for the access.logs and can rotate them even without an explicit command from Squid. For example; the log_file_daemon bundled with the latest Squid will rotate access.log on 32-bit systems whenever the current file exceeds 32-bit file sizes and starts getting errors. On the other hand the daemon helper is a separate helper partially so you can write your own and have it do whatever you like. If you do extend the bundled file daemon please consider submitting the patch for communal benefit. Also, why exactly are you asking for this? perhapse it does not exist yet because what you are needing is better or easier done some other way. Sometimes when rotating a heavy loaded (200 requests per seconds) squid with workers the cache reloads and one of the workers stops working
