On Mon, 7 Oct 2002, Shachar Shemesh wrote:
> Anyone up to writing a log analyzer for HTTP and FTP that will give us
> most accessed over past day, week, month and half year? Anything that
> only becomes frequently accessed one year back is probably not worth
> keeping around.
i already wrote something for parsing _ftp_ xferstats a while back - i
enhanced it now to generate an html table in its output, and to count
traffic per sub-directory. attached are the (html) results for running it
on the current log file.
1st report is from june 28 until october 7th.
2nd report is for all september + october's first week.
3rd report is for october's first week only.
note that the report for each directory includes anything found in it
_recursively_, and that there are parallels in the given reports - on
purpose.
the script reads the names of the directories from a config file, so
adding any more far-grained data is very easy - i just added the
sub-directories that seemed interesting to me.
some interesting remarks:
1. people still use older '.0' distributions, updating single packages.
2. most people download large files in many small chunks - i got an idea
this is how various (unfriendly?) download managers work. by small chunks,
i mean few MB per chunk when downloading an ISO image, for example. this
makes the number of accesses parameter useless - the number of unique IPs
seems more reliable, and the transfered traffic - even more.
the question is - when deciding what to keep mirrorins and which
mirrors to drop - should we optimized for ammount of data transfered (thus
favouring users with heavy downloads) or number of people making transfers
(thus favoring mirrors that are used by more users).
just for an idea regarding how much disk space various mirrors occupy:
cpan - 1,350,036KB (1.3GB)
debian - 21,459,184 (21GB)
debian-Non-US - 319,784 (0.3GB)
gnu - 4,708,192 (4.7GB)
kde - 7,294,992 (7.2GB)
kernel.org - 9,581,616 (9.5GB)
ldp - 538,408 (0.5GB)
loki - 3,246,244 (3.2GB)
mandrake - 5,755,052 (5.7GB)
mandrake-crypto - 4 (0GB)
redhat - 10,894,826 (10.8GB)
slackware - 673,936 (0.6GB)
btw, i forgot checking loki's statistics in the above reports, as well as
the non-US debain and the crypto of mandrake. :0
--
guy
"For world domination - press 1,
or dial 0, and please hold, for the creator." -- nob o. dy
Title: IGLU FTP Download Statistics
IGLU FTP Download Statistics
Directory statistics:
| Path |
Num Accesses |
KBytes |
Num Uniq. Addr. |
| /storage/ftp/pub/Hebrew |
18270 |
10,070,302 |
907 |
| /storage/ftp/pub/mirrors/kernel.org |
18378 |
20,893,013 |
425 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v1.0 |
4 |
2,481 |
4 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v1.1 |
|
0 |
0 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v1.2 |
1 |
1,648 |
1 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v1.3 |
2 |
3,341 |
1 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v2.0 |
4 |
13,159 |
4 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v2.1 |
1 |
5,888 |
1 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v2.2 |
1050 |
508,681 |
18 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v2.3 |
5 |
52,289 |
4 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v2.4 |
556 |
6,792,377 |
232 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v2.5 |
631 |
9,593,097 |
101 |
| /storage/ftp/pub/mirrors/redhat |
153747 |
1,532,261,843 |
3309 |
| /storage/ftp/pub/mirrors/redhat/updates.redhat.com/7.0 |
888 |
1,323,374 |
13 |
| /storage/ftp/pub/mirrors/redhat/updates.redhat.com/7.1 |
992 |
1,652,511 |
19 |
| /storage/ftp/pub/mirrors/redhat/updates.redhat.com/7.2 |
6121 |
7,573,973 |
70 |
| /storage/ftp/pub/mirrors/redhat/updates.redhat.com/7.3 |
6743 |
16,701,201 |
181 |
| /storage/ftp/pub/mirrors/debian |
140643 |
66,362,434 |
2107 |
| /storage/ftp/pub/mirrors/slackware |
1351 |
74,064,655 |
267 |
| /storage/ftp/pub/mirrors/mandrake |
150855 |
831,315,375 |
5968 |
| /storage/ftp/pub/mirrors/mandrake/updates/8.0 |
688 |
1,119,699 |
7 |
| /storage/ftp/pub/mirrors/mandrake/updates/8.1 |
1431 |
2,690,903 |
31 |
| /storage/ftp/pub/mirrors/mandrake/updates/8.2 |
4682 |
7,623,417 |
131 |
| /storage/ftp/pub/mirrors/mandrake/updates/9.0 |
75 |
198 |
38 |
| /storage/ftp/pub/mirrors/kde |
22923 |
40,091,991 |
688 |
| /storage/ftp/pub/mirrors/kde/stable/koffice-1.2 |
174 |
405,733 |
54 |
| /storage/ftp/pub/mirrors/kde/unstable/kde-3.1-beta2 |
92 |
66,960 |
1 |
| /storage/ftp/pub/mirrors/cpan |
6406 |
672,472 |
239 |
| /storage/ftp/pub/mirrors/gnu |
9772 |
4,302,547 |
124 |
| /storage/ftp/pub/mirrors/ldp |
8921 |
66,268 |
33 |
| /storage/ftp/pub/mirrors/linux |
|
0 |
0 |
Title: IGLU FTP Download Statistics
IGLU FTP Download Statistics
Directory statistics:
| Path |
Num Accesses |
KBytes |
Num Uniq. Addr. |
| /storage/ftp/pub/Hebrew |
3735 |
2,975,804 |
336 |
| /storage/ftp/pub/mirrors/kernel.org |
10726 |
6,138,982 |
173 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v1.0 |
1 |
1,229 |
1 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v1.1 |
|
0 |
0 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v1.2 |
1 |
1,648 |
1 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v1.3 |
2 |
3,341 |
1 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v2.0 |
1 |
0 |
1 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v2.1 |
|
0 |
0 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v2.2 |
1029 |
318,833 |
10 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v2.3 |
1 |
20,392 |
1 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v2.4 |
212 |
2,653,397 |
99 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v2.5 |
113 |
1,066,229 |
37 |
| /storage/ftp/pub/mirrors/redhat |
51165 |
486,630,501 |
1229 |
| /storage/ftp/pub/mirrors/redhat/updates.redhat.com/7.0 |
268 |
400,502 |
5 |
| /storage/ftp/pub/mirrors/redhat/updates.redhat.com/7.1 |
69 |
135,382 |
8 |
| /storage/ftp/pub/mirrors/redhat/updates.redhat.com/7.2 |
2594 |
1,978,457 |
31 |
| /storage/ftp/pub/mirrors/redhat/updates.redhat.com/7.3 |
3828 |
8,775,316 |
86 |
| /storage/ftp/pub/mirrors/debian |
49409 |
22,521,833 |
886 |
| /storage/ftp/pub/mirrors/slackware |
581 |
31,594,729 |
115 |
| /storage/ftp/pub/mirrors/mandrake |
39333 |
333,490,277 |
1909 |
| /storage/ftp/pub/mirrors/mandrake/updates/8.0 |
417 |
703,462 |
3 |
| /storage/ftp/pub/mirrors/mandrake/updates/8.1 |
1074 |
1,980,541 |
10 |
| /storage/ftp/pub/mirrors/mandrake/updates/8.2 |
1240 |
2,670,177 |
61 |
| /storage/ftp/pub/mirrors/mandrake/updates/9.0 |
63 |
176 |
29 |
| /storage/ftp/pub/mirrors/kde |
5523 |
7,498,999 |
207 |
| /storage/ftp/pub/mirrors/kde/stable/koffice-1.2 |
174 |
405,733 |
54 |
| /storage/ftp/pub/mirrors/kde/unstable/kde-3.1-beta2 |
92 |
66,960 |
1 |
| /storage/ftp/pub/mirrors/cpan |
2851 |
299,123 |
100 |
| /storage/ftp/pub/mirrors/gnu |
4878 |
814,175 |
51 |
| /storage/ftp/pub/mirrors/ldp |
444 |
3,546 |
12 |
| /storage/ftp/pub/mirrors/linux |
|
0 |
0 |
Title: IGLU FTP Download Statistics
IGLU FTP Download Statistics
Directory statistics:
| Path |
Num Accesses |
KBytes |
Num Uniq. Addr. |
| /storage/ftp/pub/Hebrew |
571 |
686,489 |
69 |
| /storage/ftp/pub/mirrors/kernel.org |
8174 |
2,997,323 |
56 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v1.0 |
|
0 |
0 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v1.1 |
|
0 |
0 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v1.2 |
|
0 |
0 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v1.3 |
|
0 |
0 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v2.0 |
|
0 |
0 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v2.1 |
|
0 |
0 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v2.2 |
1012 |
161,072 |
4 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v2.3 |
|
0 |
0 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v2.4 |
69 |
908,235 |
32 |
| /storage/ftp/pub/mirrors/kernel.org/linux/kernel/v2.5 |
19 |
296,406 |
11 |
| /storage/ftp/pub/mirrors/redhat |
15275 |
73,440,700 |
309 |
| /storage/ftp/pub/mirrors/redhat/updates.redhat.com/7.0 |
3 |
19,927 |
3 |
| /storage/ftp/pub/mirrors/redhat/updates.redhat.com/7.1 |
45 |
109,408 |
3 |
| /storage/ftp/pub/mirrors/redhat/updates.redhat.com/7.2 |
1495 |
432,749 |
9 |
| /storage/ftp/pub/mirrors/redhat/updates.redhat.com/7.3 |
557 |
1,583,416 |
27 |
| /storage/ftp/pub/mirrors/debian |
9295 |
3,985,121 |
237 |
| /storage/ftp/pub/mirrors/slackware |
79 |
8,143,863 |
28 |
| /storage/ftp/pub/mirrors/mandrake |
10576 |
194,685,005 |
615 |
| /storage/ftp/pub/mirrors/mandrake/updates/8.0 |
357 |
631,412 |
1 |
| /storage/ftp/pub/mirrors/mandrake/updates/8.1 |
361 |
743,564 |
1 |
| /storage/ftp/pub/mirrors/mandrake/updates/8.2 |
243 |
555,184 |
9 |
| /storage/ftp/pub/mirrors/mandrake/updates/9.0 |
30 |
104 |
14 |
| /storage/ftp/pub/mirrors/kde |
272 |
442,578 |
31 |
| /storage/ftp/pub/mirrors/kde/stable/koffice-1.2 |
40 |
43,707 |
10 |
| /storage/ftp/pub/mirrors/kde/unstable/kde-3.1-beta2 |
92 |
66,960 |
1 |
| /storage/ftp/pub/mirrors/cpan |
256 |
13,498 |
17 |
| /storage/ftp/pub/mirrors/gnu |
65 |
42,912 |
9 |
| /storage/ftp/pub/mirrors/ldp |
129 |
722 |
3 |
| /storage/ftp/pub/mirrors/linux |
|
0 |
0 |
