Re: [pfSense] Squid not logging traffic
Hi, try ntop package 2015-02-16 14:15 GMT-03:00 Brian Caouette bri...@dlois.com: I also notice it doesn't log torrents. Is there a way to tell it to log everything so I can get an accurate picture of what each device on the network is using? Sent from my iPad On Feb 15, 2015, at 11:09 PM, Volker Kuhlmann list0...@paradise.net.nz wrote: On Mon 16 Feb 2015 03:53:55 NZDT +1300, Brian Caouette wrote: I just noticed squid is not logging all traffic. The last few nights I've used plex on my roku connected to my friends server. The only thing showing in light squid Are you talking about squid or light squid? Aren't they different packages? Squid logs the number of bytes transferred, which means it can write the log entry only after the connection is closed the time stamps seems to be the one of when the log entry was written, not when the connection was opened. When is a streaming connection closed? Perhaps more to the point, what port does the stream use? Is it one handled by squid in the first place? Volker -- Volker Kuhlmann http://volker.top.geek.nz/Please do not CC list postings to me. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold -- Luis G. Coralle ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] Squid not logging traffic
Torrents wouldn't be tracked. They are going over a non HTTP connection. If you want to check the connection, BandwithD might be what your looking for. --Tiernan -Original Message- From: List [mailto:list-boun...@lists.pfsense.org] On Behalf Of Brian Caouette Sent: Monday 16 February 2015 17:17 To: pfSense Support and Discussion Mailing List Subject: Re: [pfSense] Squid not logging traffic bbs.dlois.com:/lightsquid/day_detail.cgi?year=2015month=02day=16 Dell wired and Roku are the busiest devices yet report almost no traffic. Sent from my iPad On Feb 15, 2015, at 11:09 PM, Volker Kuhlmann list0...@paradise.net.nz wrote: On Mon 16 Feb 2015 03:53:55 NZDT +1300, Brian Caouette wrote: I just noticed squid is not logging all traffic. The last few nights I've used plex on my roku connected to my friends server. The only thing showing in light squid Are you talking about squid or light squid? Aren't they different packages? Squid logs the number of bytes transferred, which means it can write the log entry only after the connection is closed the time stamps seems to be the one of when the log entry was written, not when the connection was opened. When is a streaming connection closed? Perhaps more to the point, what port does the stream use? Is it one handled by squid in the first place? Volker -- Volker Kuhlmann http://volker.top.geek.nz/Please do not CC list postings to me. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] Squid not logging traffic
bbs.dlois.com:/lightsquid/day_detail.cgi?year=2015month=02day=16 Dell wired and Roku are the busiest devices yet report almost no traffic. Sent from my iPad On Feb 15, 2015, at 11:09 PM, Volker Kuhlmann list0...@paradise.net.nz wrote: On Mon 16 Feb 2015 03:53:55 NZDT +1300, Brian Caouette wrote: I just noticed squid is not logging all traffic. The last few nights I've used plex on my roku connected to my friends server. The only thing showing in light squid Are you talking about squid or light squid? Aren't they different packages? Squid logs the number of bytes transferred, which means it can write the log entry only after the connection is closed the time stamps seems to be the one of when the log entry was written, not when the connection was opened. When is a streaming connection closed? Perhaps more to the point, what port does the stream use? Is it one handled by squid in the first place? Volker -- Volker Kuhlmann http://volker.top.geek.nz/Please do not CC list postings to me. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] Squid not logging traffic
I also notice it doesn't log torrents. Is there a way to tell it to log everything so I can get an accurate picture of what each device on the network is using? Sent from my iPad On Feb 15, 2015, at 11:09 PM, Volker Kuhlmann list0...@paradise.net.nz wrote: On Mon 16 Feb 2015 03:53:55 NZDT +1300, Brian Caouette wrote: I just noticed squid is not logging all traffic. The last few nights I've used plex on my roku connected to my friends server. The only thing showing in light squid Are you talking about squid or light squid? Aren't they different packages? Squid logs the number of bytes transferred, which means it can write the log entry only after the connection is closed the time stamps seems to be the one of when the log entry was written, not when the connection was opened. When is a streaming connection closed? Perhaps more to the point, what port does the stream use? Is it one handled by squid in the first place? Volker -- Volker Kuhlmann http://volker.top.geek.nz/Please do not CC list postings to me. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] Squid not logging traffic
On Tue 17 Feb 2015 10:33:21 NZDT +1300, Walter Parker wrote: In Realtime, you can use the dashboard app. The pfsense dashboard? I don't think so. traffic going through a particular interface is not so interesting. For plugins, BandwidthD and Darkstat have some information. Unfortuntely the info is of no value. I am not interested in any traffic volume between LAN, DMZ, WIFI, LAN2, etc. I am only interested in the traffic going through WAN, and with which *internal* host. The above packages can only tell me which *Internet* sites had how much traffic through WAN, but that side of the connection is of no interest to me. I want to know which of my clients have created the traffic for which I have to pay my ISP, so I can work out which flatmate has to pay for it, or fix the computer with a problem that wastes my money. I realise those in the USA and a few other countries don't have this problem, but it sure exists where I live and I'm sure it's not the only country. In any case it's good to know what gobbles up resources, even if they're free. I've used netflow on other systems to get this sort of information, but for pfSense you would have to setup a second box that ran the netflow visualizer to see the traffic information from one of the netflow plugins. Copying a file onto another computer to look at its content isn't too much of a problem. Do you know of a good tutorial that lists the software needed, and basic config for each part? Thanks, Volker -- Volker Kuhlmann http://volker.top.geek.nz/ Please do not CC list postings to me. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] Squid not logging traffic
On Tue 17 Feb 2015 06:15:46 NZDT +1300, Brian Caouette wrote: I also notice it doesn't log torrents. Is there a way to tell it to log everything I don't know about lightsquid. Squid is a web cache and I'm not sure it is even able to deal with anything but http. If you look at its config file you see that it only deals with a short list of ports in the first place, and is not involved in the rest at all. You are looking for an application filter (like squid is for http). pfsense is mainly a packet filter, those packages are already add-ons. so I can get an accurate picture of what each device on the network is using? With pfsense, short answer: no. This is my longest standing problem with pfsense. It is not able to tell me which LAN device caused how much WAN traffic. There may be half a dozen different add-on packages but all are of no use here (for different reasons). I'd really like to hear that I missed something... Volker -- Volker Kuhlmann is list0570 with the domain in header. http://volker.top.geek.nz/ Please do not CC list postings to me. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] Squid not logging traffic
I'd recommend doing it on a second box (Or turn it into a pfSense package). On Mon, Feb 16, 2015 at 3:48 PM, Brian Caouette bri...@dlois.com wrote: I looked at cacti a few days ago. It looks real nice but I have no clue how to set this up on the pfSense box. Sent from my iPad On Feb 16, 2015, at 6:27 PM, Walter Parker walt...@gmail.com wrote: For the real time monitor, if you switch from WAN to LAN, you can see who is doing spikes. For the other items, you can see how much bandwidth each internal IP addresses has used in one of those packages. Unless you have servers in a DMZ outside of the firewall or are doing some sort of traffic reflection to internal hosts, all traffic to/from a desktop to the firewall is traffic to the internet. I might do some screenshots to show what I mean (if I can find the time). For netflow, I setup a Windows application in a VM (from ManageEngine I think). It had simple instructions to tell the netflow generator (the firewall) to send the stats traffic to the Windows box. Then I used the the reporting features in the application to view how much data each host was sending/receiving. I was able to tell that one web server had way to much traffic and that a music streaming server was running 800% of normal. I understand that there are open source versions of this program that run on Linux/FreeBSD. Setting one of these up is on my todo list. With a bit of programming, I'm sure you do this with Cacti/RRD, but then again, I've been a perl programmer for 20 years, so my idea of a bit of programming might radically differ from yours :) If I can find the time, I'll see if I can find any notes. Walter On Mon, Feb 16, 2015 at 2:58 PM, Volker Kuhlmann list0...@paradise.net.nz wrote: On Tue 17 Feb 2015 10:33:21 NZDT +1300, Walter Parker wrote: In Realtime, you can use the dashboard app. The pfsense dashboard? I don't think so. traffic going through a particular interface is not so interesting. For plugins, BandwidthD and Darkstat have some information. Unfortuntely the info is of no value. I am not interested in any traffic volume between LAN, DMZ, WIFI, LAN2, etc. I am only interested in the traffic going through WAN, and with which *internal* host. The above packages can only tell me which *Internet* sites had how much traffic through WAN, but that side of the connection is of no interest to me. I want to know which of my clients have created the traffic for which I have to pay my ISP, so I can work out which flatmate has to pay for it, or fix the computer with a problem that wastes my money. I realise those in the USA and a few other countries don't have this problem, but it sure exists where I live and I'm sure it's not the only country. In any case it's good to know what gobbles up resources, even if they're free. I've used netflow on other systems to get this sort of information, but for pfSense you would have to setup a second box that ran the netflow visualizer to see the traffic information from one of the netflow plugins. Copying a file onto another computer to look at its content isn't too much of a problem. Do you know of a good tutorial that lists the software needed, and basic config for each part? Thanks, Volker -- Volker Kuhlmann http://volker.top.geek.nz/ Please do not CC list postings to me. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold -- The greatest dangers to liberty lurk in insidious encroachment by men of zeal, well-meaning but without understanding. -- Justice Louis D. Brandeis ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold -- The greatest dangers to liberty lurk in insidious encroachment by men of zeal, well-meaning but without understanding. -- Justice Louis D. Brandeis ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] Squid not logging traffic
On Tue 17 Feb 2015 12:27:34 NZDT +1300, Walter Parker wrote: For the real time monitor, if you switch from WAN to LAN, you can see who is doing spikes. For the other items, you can see how much bandwidth each internal IP addresses has used in one of those packages. Unless you have servers in a DMZ outside of the firewall or are doing some sort of traffic reflection to internal hosts, all traffic to/from a desktop to the firewall is traffic to the internet. We probably have a different idea of network topology. E.g. the wifi is on a different network (I don't trust wireless) to the LAN. Then I grab a laptop, connect it to wifi, and transfer 1GB with a desktop, LAN fileserver, or whatever. All this traffic goes through pfsense, but not through WAN, and is of no interest in finding out which LAN/wifi/etc host had how much traffic to the Internet (through WAN). bytes/s is of not much interest to me either, total bytes per day/week/month is. The problem with the pfsense bandwidth packages (all of them) is that they're interface based. They tell me how much traffic each host connected to interface A contributed to the traffic through A. What I want to know is how much traffic each host connected to interface A, B, C contributes to traffic through *D*. This is of interest to anyone charged by volume by their ISP. The netflow setup looks like the only contender for this, but it does nothing by itself and the whole setup looks a bit involved. I'll make another effort when I get the time. Open source on Linux only for me though, unless it is on pfsense. Thanks for thinking of the screenshots but I don't think they'd add much to your description. Volker -- Volker Kuhlmann http://volker.top.geek.nz/ Please do not CC list postings to me. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] Squid not logging traffic
I looked at cacti a few days ago. It looks real nice but I have no clue how to set this up on the pfSense box. Sent from my iPad On Feb 16, 2015, at 6:27 PM, Walter Parker walt...@gmail.com wrote: For the real time monitor, if you switch from WAN to LAN, you can see who is doing spikes. For the other items, you can see how much bandwidth each internal IP addresses has used in one of those packages. Unless you have servers in a DMZ outside of the firewall or are doing some sort of traffic reflection to internal hosts, all traffic to/from a desktop to the firewall is traffic to the internet. I might do some screenshots to show what I mean (if I can find the time). For netflow, I setup a Windows application in a VM (from ManageEngine I think). It had simple instructions to tell the netflow generator (the firewall) to send the stats traffic to the Windows box. Then I used the the reporting features in the application to view how much data each host was sending/receiving. I was able to tell that one web server had way to much traffic and that a music streaming server was running 800% of normal. I understand that there are open source versions of this program that run on Linux/FreeBSD. Setting one of these up is on my todo list. With a bit of programming, I'm sure you do this with Cacti/RRD, but then again, I've been a perl programmer for 20 years, so my idea of a bit of programming might radically differ from yours :) If I can find the time, I'll see if I can find any notes. Walter On Mon, Feb 16, 2015 at 2:58 PM, Volker Kuhlmann list0...@paradise.net.nz wrote: On Tue 17 Feb 2015 10:33:21 NZDT +1300, Walter Parker wrote: In Realtime, you can use the dashboard app. The pfsense dashboard? I don't think so. traffic going through a particular interface is not so interesting. For plugins, BandwidthD and Darkstat have some information. Unfortuntely the info is of no value. I am not interested in any traffic volume between LAN, DMZ, WIFI, LAN2, etc. I am only interested in the traffic going through WAN, and with which *internal* host. The above packages can only tell me which *Internet* sites had how much traffic through WAN, but that side of the connection is of no interest to me. I want to know which of my clients have created the traffic for which I have to pay my ISP, so I can work out which flatmate has to pay for it, or fix the computer with a problem that wastes my money. I realise those in the USA and a few other countries don't have this problem, but it sure exists where I live and I'm sure it's not the only country. In any case it's good to know what gobbles up resources, even if they're free. I've used netflow on other systems to get this sort of information, but for pfSense you would have to setup a second box that ran the netflow visualizer to see the traffic information from one of the netflow plugins. Copying a file onto another computer to look at its content isn't too much of a problem. Do you know of a good tutorial that lists the software needed, and basic config for each part? Thanks, Volker -- Volker Kuhlmann http://volker.top.geek.nz/ Please do not CC list postings to me. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold -- The greatest dangers to liberty lurk in insidious encroachment by men of zeal, well-meaning but without understanding. -- Justice Louis D. Brandeis ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] Squid not logging traffic
For the real time monitor, if you switch from WAN to LAN, you can see who is doing spikes. For the other items, you can see how much bandwidth each internal IP addresses has used in one of those packages. Unless you have servers in a DMZ outside of the firewall or are doing some sort of traffic reflection to internal hosts, all traffic to/from a desktop to the firewall is traffic to the internet. I might do some screenshots to show what I mean (if I can find the time). For netflow, I setup a Windows application in a VM (from ManageEngine I think). It had simple instructions to tell the netflow generator (the firewall) to send the stats traffic to the Windows box. Then I used the the reporting features in the application to view how much data each host was sending/receiving. I was able to tell that one web server had way to much traffic and that a music streaming server was running 800% of normal. I understand that there are open source versions of this program that run on Linux/FreeBSD. Setting one of these up is on my todo list. With a bit of programming, I'm sure you do this with Cacti/RRD, but then again, I've been a perl programmer for 20 years, so my idea of a bit of programming might radically differ from yours :) If I can find the time, I'll see if I can find any notes. Walter On Mon, Feb 16, 2015 at 2:58 PM, Volker Kuhlmann list0...@paradise.net.nz wrote: On Tue 17 Feb 2015 10:33:21 NZDT +1300, Walter Parker wrote: In Realtime, you can use the dashboard app. The pfsense dashboard? I don't think so. traffic going through a particular interface is not so interesting. For plugins, BandwidthD and Darkstat have some information. Unfortuntely the info is of no value. I am not interested in any traffic volume between LAN, DMZ, WIFI, LAN2, etc. I am only interested in the traffic going through WAN, and with which *internal* host. The above packages can only tell me which *Internet* sites had how much traffic through WAN, but that side of the connection is of no interest to me. I want to know which of my clients have created the traffic for which I have to pay my ISP, so I can work out which flatmate has to pay for it, or fix the computer with a problem that wastes my money. I realise those in the USA and a few other countries don't have this problem, but it sure exists where I live and I'm sure it's not the only country. In any case it's good to know what gobbles up resources, even if they're free. I've used netflow on other systems to get this sort of information, but for pfSense you would have to setup a second box that ran the netflow visualizer to see the traffic information from one of the netflow plugins. Copying a file onto another computer to look at its content isn't too much of a problem. Do you know of a good tutorial that lists the software needed, and basic config for each part? Thanks, Volker -- Volker Kuhlmann http://volker.top.geek.nz/ Please do not CC list postings to me. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold -- The greatest dangers to liberty lurk in insidious encroachment by men of zeal, well-meaning but without understanding. -- Justice Louis D. Brandeis ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] Squid not logging traffic
In Realtime, you can use the dashboard app. For plugins, BandwidthD and Darkstat have some information. I've used netflow on other systems to get this sort of information, but for pfSense you would have to setup a second box that ran the netflow visualizer to see the traffic information from one of the netflow plugins. On Mon, Feb 16, 2015 at 1:13 PM, Volker Kuhlmann list0...@paradise.net.nz wrote: On Tue 17 Feb 2015 06:15:46 NZDT +1300, Brian Caouette wrote: I also notice it doesn't log torrents. Is there a way to tell it to log everything I don't know about lightsquid. Squid is a web cache and I'm not sure it is even able to deal with anything but http. If you look at its config file you see that it only deals with a short list of ports in the first place, and is not involved in the rest at all. You are looking for an application filter (like squid is for http). pfsense is mainly a packet filter, those packages are already add-ons. so I can get an accurate picture of what each device on the network is using? With pfsense, short answer: no. This is my longest standing problem with pfsense. It is not able to tell me which LAN device caused how much WAN traffic. There may be half a dozen different add-on packages but all are of no use here (for different reasons). I'd really like to hear that I missed something... Volker -- Volker Kuhlmann is list0570 with the domain in header. http://volker.top.geek.nz/ Please do not CC list postings to me. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold -- The greatest dangers to liberty lurk in insidious encroachment by men of zeal, well-meaning but without understanding. -- Justice Louis D. Brandeis ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
[pfSense] Squid not logging traffic
I just noticed squid is not logging all traffic. The last few nights I've used plex on my roku connected to my friends server. The only thing showing in light squid for my roku is the calls home to the roku domain. Not of the traffic from streaming movies all night was recorded. I believe netflix does the same thing. Am i missing something? If i watch the pfsense dashboard I can see the traffic on the rrd graphs. Sent from my U.S. Cellular® Smartphone ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
Re: [pfSense] Squid not logging traffic
On Mon 16 Feb 2015 03:53:55 NZDT +1300, Brian Caouette wrote: I just noticed squid is not logging all traffic. The last few nights I've used plex on my roku connected to my friends server. The only thing showing in light squid Are you talking about squid or light squid? Aren't they different packages? Squid logs the number of bytes transferred, which means it can write the log entry only after the connection is closed the time stamps seems to be the one of when the log entry was written, not when the connection was opened. When is a streaming connection closed? Perhaps more to the point, what port does the stream use? Is it one handled by squid in the first place? Volker -- Volker Kuhlmann http://volker.top.geek.nz/ Please do not CC list postings to me. ___ pfSense mailing list https://lists.pfsense.org/mailman/listinfo/list Support the project with Gold! https://pfsense.org/gold
