tried that. :-(
I just dont have the time to sit and resolve dependencies.
why cant it be packaged together.
I would rather spend my time doing my code rather than try this.
-bash-4.1# rpm -i rsyslog-7.2.7-1.el6.x86_64.rpm
warning: rsyslog-7.2.7-1.el6.x86_64.rpm: Header V3 RSA/SHA1 Signature, key
ID e00b8985: NOKEY
error: Failed dependencies:
libee.so.0()(64bit) is needed by rsyslog-7.2.7-1.el6.x86_64
libestr.so.0()(64bit) is needed by rsyslog-7.2.7-1.el6.x86_64
liblognorm.so.0()(64bit) is needed by rsyslog-7.2.7-1.el6.x86_64
-bash-4.1#
-bash-4.1# ls /usr/lib/libestr.so.0
/usr/lib/libestr.so.0
-bash-4.1# ls /usr/lib/libee.so.0
/usr/lib/libee.so.0
-bash-4.1# rpm -i liblognorm-0.3.5-1.fc19.i686.rpm
warning: liblognorm-0.3.5-1.fc19.i686.rpm: Header V3 RSA/SHA256 Signature,
key ID fb4b18e6: NOKEY
error: Failed dependencies:
libee.so.0 is needed by liblognorm-0.3.5-1.fc19.i686
libestr.so.0 is needed by liblognorm-0.3.5-1.fc19.i686
On Fri, Jun 14, 2013 at 4:27 PM, Rainer Gerhards
<[email protected]>wrote:
> On Fri, Jun 14, 2013 at 12:55 PM, Mahesh V
> <[email protected]>wrote:
>
> > Hi David,
> > I tried compiling 7.4.0 but it is a major pain.
> > Even after installing all the requisites, I get the J-SON error and EE
> > error etc.
> > Do you have a readymade configure file which I can use by any chance ?
> > (which takes care of the
> > J-SON workaround etc)?
> >
>
> for many platforms, they are available on http://www.rsyslog.com - just
> klick on the "project" menu item.
>
> HTH
> Rainer
>
> > thanks
> > Mahesh
> >
> >
> > On Fri, Jun 14, 2013 at 4:22 PM, Rainer Gerhards
> > <[email protected]>wrote:
> >
> > > I have checked the ChangeLog, mysql batching support was added in
> 7.3.5.
> > So
> > > 7.4.0 should give a major boost in performance.
> > >
> > > Rainer
> > >
> > >
> > > On Fri, Jun 14, 2013 at 11:23 AM, Mahesh V
> > > <[email protected]>wrote:
> > >
> > > > Hi david,
> > > >
> > > > Here is the data. for operations on rsyslog + mysql and only rsyslog
> > > >
> > > > #include <syslog.h>
> > > > #include <unistd.h>
> > > > void main()
> > > > {
> > > > int i = 0;
> > > > setlogmask (LOG_UPTO (LOG_NOTICE));
> > > > openlog ("exampleprog", LOG_PID | LOG_NDELAY, LOG_DAEMON);
> > > > for(i = 0; i < 50000; i++) {
> > > > syslog (LOG_NOTICE, "APSIM A tree falls in a forest
> > %d",
> > > > i);
> > > > usleep(200);
> > > > }
> > > > closelog ();
> > > > }
> > > >
> > > > 1) rsyslog + mysql 50k entries
> > > > ----------------------------
> > > > bash-4.1# date; ./a.out; date
> > > > Tue Jun 11 11:06:48 IST 2013
> > > > Tue Jun 11 11:14:04 IST 2013
> > > >
> > > > + another minute more to insert it into mysql
> > > > and to query - took about 3 seconds to get all 50k rows.
> > > >
> > > > So rsyslog + mysql took about (8 + 1) minutes to complete
> > > >
> > > > 2) only rsyslog (writes to /var/log/127.0.0.1/syslog.log)
> > > > ------------------------------------------------------------------
> > > > bash-4.1# date; ./a.out; date
> > > > Tue Jun 11 11:17:42 IST 2013
> > > > Tue Jun 11 11:17:55 IST 2013
> > > >
> > > > Inserting only in rsyslog took about 13 seconds.
> > > >
> > > > bash-4.1# cat /var/log/127.0.0.1/syslog.log
> > > > Jun 11 11:17:55 localhost exampleprog[16191]: APSIM A tree falls in a
> > > > forest 49998
> > > > Jun 11 11:17:55 localhost exampleprog[16191]: APSIM A tree falls in a
> > > > forest 49999
> > > >
> > > > bash-4.1# date; cat /var/log/127.0.0.1/syslog.log | wc -l; date;
> > > > Tue Jun 11 11:18:43 IST 2013
> > > > 50008
> > > > Tue Jun 11 11:18:43 IST 2013
> > > >
> > > > I know that flat file insert is much faster than mysql, however,
> mysql
> > > > provides querries which will
> > > > make my life a lot easier to debug things.
> > > >
> > > > I am contemplating between using only rsyslog and then extract the
> > > contents
> > > > myself (parsing the file
> > > > using a script after the whole operation is complete) vis-a-vis
> > rsyslog +
> > > > mysql to insert logs directly
> > > > so that I can query later.
> > > >
> > > > The former seems to be the correct option for now.
> > > >
> > > > Kindly advise if anything else can be done.
> > > >
> > > > thanks
> > > > Mahesh
> > > >
> > > >
> > > >
> > > > On Fri, Jun 14, 2013 at 8:31 AM, David Lang <[email protected]> wrote:
> > > >
> > > > > does this mean that it's now working for you at high rates?
> > > > >
> > > > > David Lang
> > > > >
> > > > >
> > > > > On Fri, 14 Jun 2013, Mahesh V wrote:
> > > > >
> > > > > Thanks a lot David and all.
> > > > >>
> > > > >> regards
> > > > >> Mahesh
> > > > >>
> > > > >>
> > > > >> On Fri, Jun 14, 2013 at 1:36 AM, David Lang <[email protected]>
> wrote:
> > > > >>
> > > > >> On Thu, 13 Jun 2013, Mahesh V wrote:
> > > > >>>
> > > > >>> Hello David,
> > > > >>>
> > > > >>>>
> > > > >>>> Thanks for the reply.
> > > > >>>>
> > > > >>>> Here is another set of data points.
> > > > >>>> A simple program (shamelessly copied from internet) which does
> > > nothing
> > > > >>>> but
> > > > >>>> inserts in mysql through rsyslog.
> > > > >>>>
> > > > >>>> #include <syslog.h>
> > > > >>>> #include <unistd.h>
> > > > >>>> void main()
> > > > >>>> {
> > > > >>>> int i = 0;
> > > > >>>> setlogmask (LOG_UPTO (LOG_NOTICE));
> > > > >>>> openlog ("exampleprog", LOG_PID | LOG_NDELAY, LOG_DAEMON);
> > > > >>>> for(i = 0; i < 10000; i++) {
> > > > >>>> syslog (LOG_NOTICE, "A tree falls in a forest %d", i);
> > > > >>>> usleep(100);
> > > > >>>> }
> > > > >>>> closelog ();
> > > > >>>> }
> > > > >>>>
> > > > >>>> rsyslog.conf
> > > > >>>> ----------------
> > > > >>>> $SystemLogRateLimitInterval 1
> > > > >>>> $SystemLogRateLimitBurst 5000
> > > > >>>> ## Optional
> > > > >>>>
> > > > >>>> program run
> > > > >>>> -------------------
> > > > >>>> -bash-4.1# date; ./a.out; date
> > > > >>>> Mon Jun 10 13:02:10 IST 2013
> > > > >>>> Mon Jun 10 13:02:12 IST 2013
> > > > >>>>
> > > > >>>> Close to 5 minutes later
> > > > >>>> ------------------------------****--------
> > > > >>>>
> > > > >>>> mysql -u root -p
> > > > >>>> mysql> select message from SystemEvents;
> > > > >>>> ...
> > > > >>>> ...
> > > > >>>> imuxsock begins to drop messages from pid 16346 due to
> > > rate-limiting |
> > > > >>>> imuxsock lost 1383 messages from pid 16346 due to rate-limiting
> > > |
> > > > >>>> | A tree falls in a forest 9998
> > > > |
> > > > >>>> | A tree falls in a forest 9999
> > > > |
> > > > >>>>
> > +-----------------------------****----------------------------**--**
> > > > >>>>
> > > > >>>> -----------+
> > > > >>>> 8619 rows in set (0.01 sec)
> > > > >>>>
> > > > >>>> I found that out of 10000, some 8616 records were inserted in 2
> > > > seconds.
> > > > >>>> Wondering why imuxsock dropped messages even though rate
> limiting
> > > > value
> > > > >>>> was
> > > > >>>> high.
> > > > >>>>
> > > > >>>>
> > > > >>> well, you have the rate limit set to 5000 per second, but you are
> > > > sending
> > > > >>> double that within 2 seconds. It doesn't seem surprising that you
> > > could
> > > > >>> loose ~1400 of those messages.
> > > > >>>
> > > > >>> 5000 per second isn't especially high as far as rsyslog is
> > concerned.
> > > > >>>
> > > > >>> personally, I never turn on rate limiting
> > > > >>>
> > > > >>>
> > > > >>> Obviously there would be system limitation, but that should be
> the
> > > > only
> > > > >>>
> > > > >>>> criteria.
> > > > >>>>
> > > > >>>>
> > > > >>> raise the limit to something higher than the rate that you are
> > > sending
> > > > >>> logs.
> > > > >>>
> > > > >>>
> > > > >>> -bash-4.1# /sbin/rsyslogd -v
> > > > >>>
> > > > >>>> rsyslogd 5.8.10, compiled with:
> > > > >>>>
> > > > >>>> this in no way gives disk spinning capacity but on a server
> with 4
> > > > >>>> cores,
> > > > >>>> would this be the max I can reach?
> > > > >>>>
> > > > >>>>
> > > > >>> no, you just hit the max that you had configured.
> > > > >>>
> > > > >>> go to a very simple configuration, remove any rate limiting, just
> > put
> > > > in
> > > > >>> the minimum needed to read the input and output to a file. on
> your
> > 4
> > > > core
> > > > >>> box you will get hundreds of thousands, if not millions, of logs
> > per
> > > > >>> second
> > > > >>> (and really only be using 2-3 cores to do so)
> > > > >>>
> > > > >>> David Lang
> > > > >>>
> > > > >>> thanks
> > > > >>>
> > > > >>>> Mahesh
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> On Wed, Jun 12, 2013 at 9:33 PM, David Lang <[email protected]>
> > wrote:
> > > > >>>>
> > > > >>>> On Wed, 12 Jun 2013, [email protected] wrote:
> > > > >>>>
> > > > >>>>>
> > > > >>>>> On Wed, Jun 12, 2013 at 07:44:50PM +0530, Mahesh V wrote:
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>> Hello,
> > > > >>>>>>
> > > > >>>>>>>
> > > > >>>>>>> The software i work, has syslog facility enabled for debug
> > logs.
> > > > >>>>>>> It generates a huge amount of logs (12000 lines in 2
> minutes) -
> > > > about
> > > > >>>>>>> 100
> > > > >>>>>>> log entries per second and this is put into a mysql backend.
> > > > >>>>>>> However, I see that after about 2 minutes of run, my
> > application
> > > > >>>>>>> stalls
> > > > >>>>>>> and if I disable (delete the SystemEvents table in Syslog
> > > > database),
> > > > >>>>>>> my
> > > > >>>>>>> application continues fine.
> > > > >>>>>>> My my.cnf and rsyslog.conf is attached along.
> > > > >>>>>>> Any ideas?
> > > > >>>>>>>
> > > > >>>>>>> Thanks
> > > > >>>>>>> Mahesh
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>> Hi Mahesh,
> > > > >>>>>>
> > > > >>>>>> You sent a lot of software configuration information, but none
> > on
> > > > your
> > > > >>>>>> DB hardware and I/O configurations. It sounds like you do not
> > have
> > > > >>>>>> enough
> > > > >>>>>> capacity there, but this is complete speculation without the
> > > actual
> > > > >>>>>> details.
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> expanding on this, you don't provide the details of your
> disks.
> > > 100
> > > > >>>>> transactions/sec sounds like you are doing this on a single
> 7200
> > > rpm
> > > > >>>>> disk.
> > > > >>>>> Since MySQL wants to keep the data safe, it does a fsync as
> part
> > of
> > > > >>>>> each
> > > > >>>>> transaction. With a 7200 rpm disk it is doing just over 100
> > > > >>>>> rotations/sec
> > > > >>>>> and so it can't do fwyncs any faster than that. For high
> > > performance
> > > > >>>>> database work you need a disk controller card with battery
> backed
> > > > cache
> > > > >>>>> configured for writeback. That way the controller can tell the
> OS
> > > the
> > > > >>>>> write
> > > > >>>>> is finished as soon as it's in the cache (where it will get
> > > written,
> > > > >>>>> even
> > > > >>>>> if the system crashes, as long as the disks and controller get
> > > power
> > > > >>>>> again
> > > > >>>>> before the battery dies). This can change the transaction rate
> to
> > > > tens
> > > > >>>>> of
> > > > >>>>> thousands per second.
> > > > >>>>>
> > > > >>>>> In addition, rsyslog has the ability to write more than one
> > message
> > > > per
> > > > >>>>> transation (batch mode), but you need to move to rsyslog 7.4
> > (well
> > > > >>>>> 7.3.5
> > > > >>>>> or
> > > > >>>>> later). Once you do that you should experiment with the batch
> > size.
> > > > >>>>> It's
> > > > >>>>> very likely that you will want a batch size somewhere >1000
> > > > >>>>> messages/batch,
> > > > >>>>> but it will depend on how fast your disks are able to keep up.
> > > > >>>>>
> > > > >>>>> David Lang
> > > > >>>>>
> > > > >>>>> ______________________________******_________________
> > > > >>>>> rsyslog mailing list
> > > > >>>>> http://lists.adiscon.net/******mailman/listinfo/rsyslog<
> > > > http://lists.adiscon.net/****mailman/listinfo/rsyslog>
> > > > >>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<
> > > > http://lists.adiscon.net/**mailman/listinfo/rsyslog>
> > > > >>>>> >
> > > > >>>>> <http:**//lists.adiscon.net/**mailman/**listinfo/rsyslog<
> > > > http://lists.adiscon.net/mailman/**listinfo/rsyslog>
> > > > >>>>> <htt**p://lists.adiscon.net/mailman/**listinfo/rsyslog<
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog>
> > > > >>>>> >
> > > > >>>>>
> > > > >>>>>>
> > > > >>>>>> http://www.rsyslog.com/******professional-services/<
> > > > http://www.rsyslog.com/****professional-services/>
> > > > >>>>> <http://**www.rsyslog.com/****professional-services/<
> > > > http://www.rsyslog.com/**professional-services/>
> > > > >>>>> >
> > > > >>>>> <http://**www.rsyslog.com/**professional-**services/<
> > > > http://www.rsyslog.com/professional-**services/>
> > > > >>>>> <http:**//www.rsyslog.com/**professional-services/<
> > > > http://www.rsyslog.com/professional-services/>
> > > > >>>>> >
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>>
> > > > >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > > >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
> by a
> > > > >>>>> myriad
> > > > >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> > if
> > > > you
> > > > >>>>> DON'T LIKE THAT.
> > > > >>>>>
> > > > >>>>> ______________________________****_________________
> > > > >>>>>
> > > > >>>> rsyslog mailing list
> > > > >>>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<
> > > > http://lists.adiscon.net/**mailman/listinfo/rsyslog>
> > > > >>>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog>
> > > > >>>> >
> > > > >>>> http://www.rsyslog.com/****professional-services/<
> > > > http://www.rsyslog.com/**professional-services/>
> > > > >>>> <http://**www.rsyslog.com/professional-**services/<
> > > > http://www.rsyslog.com/professional-services/>
> > > > >>>> >
> > > > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
> a
> > > > myriad
> > > > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if
> > > you
> > > > >>>> DON'T LIKE THAT.
> > > > >>>>
> > > > >>>> ______________________________****_________________
> > > > >>>>
> > > > >>> rsyslog mailing list
> > > > >>> http://lists.adiscon.net/****mailman/listinfo/rsyslog<
> > > > http://lists.adiscon.net/**mailman/listinfo/rsyslog>
> > > > >>> <http:**//lists.adiscon.net/mailman/**listinfo/rsyslog<
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog>
> > > > >>> >
> > > > >>> http://www.rsyslog.com/****professional-services/<
> > > > http://www.rsyslog.com/**professional-services/>
> > > > >>> <http://**www.rsyslog.com/professional-**services/<
> > > > http://www.rsyslog.com/professional-services/>
> > > > >>> >
> > > > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > > > myriad
> > > > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if
> > > you
> > > > >>> DON'T LIKE THAT.
> > > > >>>
> > > > >>> ______________________________**_________________
> > > > >> rsyslog mailing list
> > > > >> http://lists.adiscon.net/**mailman/listinfo/rsyslog<
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog>
> > > > >> http://www.rsyslog.com/**professional-services/<
> > > > http://www.rsyslog.com/professional-services/>
> > > > >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > > myriad
> > > > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> > you
> > > > >> DON'T LIKE THAT.
> > > > >>
> > > > >> ______________________________**_________________
> > > > > rsyslog mailing list
> > > > > http://lists.adiscon.net/**mailman/listinfo/rsyslog<
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog>
> > > > > http://www.rsyslog.com/**professional-services/<
> > > > http://www.rsyslog.com/professional-services/>
> > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > > myriad
> > > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> > you
> > > > > DON'T LIKE THAT.
> > > > >
> > > > _______________________________________________
> > > > rsyslog mailing list
> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > > http://www.rsyslog.com/professional-services/
> > > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad
> > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> you
> > > > DON'T LIKE THAT.
> > > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > > DON'T LIKE THAT.
> > >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
