Kang, saya sudah uncompressed datanya menjadi 1.9GB sehingga ndak perlu
"gunzip -c" lagi. Perl-nya juga saya bikin simpel, tapi koq tetep killed
ya ? apa memang perl ndak bisa buka file yang besar ?
---perlkilled.pl---
#!/usr/bin/perl -w
use strict;
$|=1;
my $LOGF=$ARGV[0];
open(LOG,"$LOGF") || die "Can't open file $LOGF: $!\n";
foreach (<LOG>) {
print "x";
}
close (LOG);
print "finish!!\n";
---
---error---
[EMAIL PROTECTED] ~]$ ./perlkilled.pl
/var/www/html/reports/logs/uncompr_20080226.log
Killed
[EMAIL PROTECTED] ~]# free -m
total used free shared buffers cached
Mem: 2027 655 1372 0 1 599
-/+ buffers/cache: 53 1973
Swap: 511 23 488
[EMAIL PROTECTED] ~]#
---
Help!
Eko Prasetya wrote:
Darn, baca perl susah amat ya.
Anyhow, problemnya out of memory, Anda tentu sudah tahu.
Kemungkinan sih dari open() yang gunzip itu. Saya takutnya kalau gunzip() itu
dikirim ke pipe, dan open() di situ harus simpan semua datanya di buffer, 2GB
is a big number.
Kalau mau di coba, setelah open(gunzip), coba print() sesuatu. Coba cari tahu
apakah open() nya berhasil atau tidak.
Kalau misalkan tidak berhasil, mungkin sebaiknya file-nya di gunzip dulu, baru
di open() hasil uncompressed-nya, dengan asumsi hard-drive-nya masih punya
free-space banyak.
KOkon.
On Fri, Apr 18, 2008 at 2:10 AM, dv0r4k <[EMAIL PROTECTED]> wrote:
Halo semua,
Saya membuat perl script untuk ngebaca log apache terkompresi dgn gzip.
Jika skrip ini saya jalankan ke log ukuran kecil script tsb berjalan dgn
baik. Tapi begitu saya jalanin di ke file besar (100MB compressed, 2GB
uncompressed), script tsb akan terbunuh (sepertinya oleh OS (ulimit?)).
Kenapa yak ?? Mungkin cara open() saya salah ?? help!
EF
.:cnt_sssn.pl:.
======================================================
#!/usr/bin/perl -w
use strict;
use Time::Local;
$|=1;
my $SESSION=24; # to define 1 session, in
minutes.
my $LOGF=$ARGV[0];
my @uips;
my $uiplmt=500; # limit before it flush to file
my $uipflcntr = 0;
my $sesslim = $SESSION * 60;
my $totalsession;
open(LOG,"gunzip -c $LOGF | ") || die "Can't open file $LOGF: $!\n";
foreach (<LOG>) {
chomp;
my @rawln = split(/\s{1,}/,$_,6);
next if (not defined($rawln[5]));
my ($ips,$timing,$url) = ($rawln[5] =~ m#^(.*) \[(.*)\] (".*)#);
push(@uips,$ips) if ( ! grep(/$ips/,@uips) );
if ($#uips > $uiplmt) {
print "UIP reached $uiplmt, flush to file
.uip.$uipflcntr.\n";
$,="\n";
open(UIPFL,">> .uip.$uipflcntr");
print UIPFL @uips;
close(UIPFL);
undef @uips;
$uipflcntr++;
}
}
close(LOG);
print "finish\n";
======================================================
Ketika di running:::
======================================================
[EMAIL PROTECTED] ~]$ ./cnt_sssn.pl /var/www/html/reports/logs/20071012.log.gz
finish
[EMAIL PROTECTED] ~]$ ./cnt_sssn.pl /var/www/html/reports/logs/20080226.log.gz
Killed
[EMAIL PROTECTED] ~]$ ll -h
/var/www/html/reports/logs/{20071012.log.gz,20080226.log.gz}
-rw-rw-r-- 1 batch devel 22K Oct 12 2007
/var/www/html/reports/logs/20071012.log.gz
-rw-rw-r-- 1 batch root 107M Feb 26 23:59
/var/www/html/reports/logs/20080226.log.gz
[EMAIL PROTECTED] ~]$ dmesg | tail
HighMem: 60*4kB 46*8kB 10*16kB 2*32kB 0*64kB 0*128kB 0*256kB 1*512kB
0*1024kB 0*2048kB 0*4096kB = 1344kB
Swap cache: add 5787810, delete 5780748, find 8492876/8794717, race 0+3
0 bounce buffer pages
Free swap: 0kB
524282 pages of RAM
294906 pages of HIGHMEM
5289 reserved pages
31741 pages shared
7062 pages swap cached
Out of Memory: Killed process 20900 (cnt_sssn.pl).
[EMAIL PROTECTED] ~]$
======================================================
contoh data (log apache):::
======================================================
Oct 12 23:51:15 10.0.0.2 ip_log: 10.0.0.5 "- -", - - [12/Oct/2007:23:50:50
+0800] "GET / HTTP/1.0" 200 8784 "-" "check_http/1.104 (nagios-plugins
1.4.9)"
Oct 12 23:53:37 10.0.0.3 ip_log: 10.0.0.5 "- -", - - [12/Oct/2007:23:53:24
+0800] "GET / HTTP/1.0" 200 8784 "-" "check_http/1.104 (nagios-plugins
1.4.9)"
Oct 12 23:54:15 10.0.0.2 ip_log: 10.0.0.5 "- -", - - [12/Oct/2007:23:53:50
+0800] "GET / HTTP/1.0" 200 8784 "-" "check_http/1.104 (nagios-plugins
1.4.9)"
======================================================
--
Berhenti langganan: [EMAIL PROTECTED]
Arsip dan info: http://linux.or.id/milis
