2013/6/14 Mahesh V <[email protected]> > Hi david, > > Here is the data. for operations on rsyslog + mysql and only rsyslog > > #include <syslog.h> > #include <unistd.h> > void main() > { > int i = 0; > setlogmask (LOG_UPTO (LOG_NOTICE)); > openlog ("exampleprog", LOG_PID | LOG_NDELAY, LOG_DAEMON); > for(i = 0; i < 50000; i++) { > syslog (LOG_NOTICE, "APSIM A tree falls in a forest %d", > i); > usleep(200); > } > closelog (); > } > > 1) rsyslog + mysql 50k entries > ---------------------------- > bash-4.1# date; ./a.out; date > Tue Jun 11 11:06:48 IST 2013 > Tue Jun 11 11:14:04 IST 2013 > > + another minute more to insert it into mysql > and to query - took about 3 seconds to get all 50k rows. > > So rsyslog + mysql took about (8 + 1) minutes to complete > > 2) only rsyslog (writes to /var/log/127.0.0.1/syslog.log) > ------------------------------------------------------------------ > bash-4.1# date; ./a.out; date > Tue Jun 11 11:17:42 IST 2013 > Tue Jun 11 11:17:55 IST 2013 > > Inserting only in rsyslog took about 13 seconds. > > bash-4.1# cat /var/log/127.0.0.1/syslog.log > Jun 11 11:17:55 localhost exampleprog[16191]: APSIM A tree falls in a > forest 49998 > Jun 11 11:17:55 localhost exampleprog[16191]: APSIM A tree falls in a > forest 49999 > > bash-4.1# date; cat /var/log/127.0.0.1/syslog.log | wc -l; date; > Tue Jun 11 11:18:43 IST 2013 > 50008 > Tue Jun 11 11:18:43 IST 2013 > > I know that flat file insert is much faster than mysql, however, mysql > provides querries which will > make my life a lot easier to debug things. > > I am contemplating between using only rsyslog and then extract the contents > myself (parsing the file > using a script after the whole operation is complete) vis-a-vis rsyslog + > mysql to insert logs directly > so that I can query later. > > The former seems to be the correct option for now. > > Kindly advise if anything else can be done. > > Hi Mahesh,
If you don't need mysql for a specific reason, I'd suggest you try thowing your logs in Elasticsearch. Here's a tutorial: http://wiki.rsyslog.com/index.php/HOWTO:_rsyslog_%2B_elasticsearch I assume you'll get way better insert and query performance than you can with mysql (ie: with bulks, I get 10-20K logs indexed per second on my $500 laptop. Then I can query in 100M-200M logs within a second. Depends on your settings). Plus, it's super-easy to scale Elasticsearch by adding new nodes. For querying, there are several, tools, the most popular being Kibana: http://three.kibana.org/ There are packages maintained by Adiscon (RHEL&Ubuntu) which include the newest stable rsyslog and the ES output module (rsyslog-elasticsearch). _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
