Hi, The cited 100k EPS does not apply to all input and output modules. This generally meant that the architecture of nxlog makes this possible. You can try it with file or tcp modules. If the bottleneck is at the other end, there is not much we can do. In this case you are sending INSERTs to mysql which require a network roundtrip for each event and for this reason it will never scale up to this range, at most you can expect is a few thousand EPS on a commodity box. Depending on the mysql engine and indexing method you use, this can affect insert speed even more , especially if you are storing a large number of events. There are various methods to overcome this, e.g. using batched loading with LOAD DATA (as is done in our enterprise edition).
Since you already wrote C code to test, others would probably also appreciate if you could identify what the problems are with om_dbi. Regards, Botond On Thu, 13 Jun 2013 17:58:05 +0300 cagil ozturk <[email protected]> wrote: > Hi All, > > nxlog feature list specifies that it can handle 100.000 events per second > (EPS); However I tried to sent 100.000 logs over tcp and write it to mysql > database and it took over a minute. I suspected that the libdbi > performance is lower first. So, I inspected libdbi mysql performance and > write a c code to insert same data, it took 7 seconds. So the problem seems > to be with nxlog server when writing to database. > > How can I configure to get the expected performance 100.000 EPS. Here is my > configuration: > > <Input in> > Module im_tcp > Host 0.0.0.0 > Port 514 > Exec parse_syslog(); > </Input> > > <Output dbi> > Exec parse_syslog_bsd(); > Module om_dbi > SQL INSERT INTO log (facility, severity, hostname, timestamp, application, > message) \ > VALUES ($SyslogFacility, $SyslogSeverity, $Hostname, '$EventTime', > $SourceName, $Message) > Driver mysql > Option host localhost > Option username admin > Option password somepasswd > Option dbname NXLOGS > </Output> ------------------------------------------------------------------------------ This SF.net email is sponsored by Windows: Build for Windows Store. http://p.sf.net/sfu/windows-dev2dev _______________________________________________ nxlog-ce-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
