I'm still working on that...I believe the smart way is to go with merge tables. Ideas? Suggestions?
On 7/10/06, Freeman, Michael <[EMAIL PROTECTED]> wrote: > Speaking of scalability.. I know cdukes dropped his previous version > which I assume was built because of some issues with the clund branch > not scaling well.. Now we are back to cdukes working on the clund > branch, and the innodb stuff is gone, is it the same or better than the > InnoDB stuff? How about when there are > 5000 hosts in the database? > > -----Original Message----- > From: Clayton Dukes [mailto:[EMAIL PROTECTED] > Sent: Monday, July 10, 2006 1:56 PM > To: Claus Lund > Cc: Freeman, Michael; php-syslog-ng-support@lists.sourceforge.net > Subject: Re: [Php-syslog-ng-support] Log2rss support? > > Agreed, we need to consider that many of us see multi-millions of > messages on these systems... > > > On 7/10/06, Claus Lund <[EMAIL PROTECTED]> wrote: > > It may be more flexible but one of my number one priorities is that it > > > has to scale very well. Depending on the number of subscribers and how > > > many thousand log entries you have per minute then a dynamic/on demand > > > solution that queries the entire database may just not work. > > > > -Claus > > > > On 7/10/06, Freeman, Michael <[EMAIL PROTECTED]> wrote: > > > I don't think you really need a separate feed, since the data > > > already exists in the database. I think it would be a little more > > > flexible if you had a .php script (renamed as .rss or something) > > > building the actual feed for you, every time you demanded it. > > > > > > -----Original Message----- > > > From: Claus Lund [mailto:[EMAIL PROTECTED] > > > Sent: Monday, July 10, 2006 1:43 PM > > > To: Freeman, Michael > > > Cc: php-syslog-ng-support@lists.sourceforge.net > > > Subject: Re: [Php-syslog-ng-support] Log2rss support? > > > > > > I think Jason probably has the right idea to somehow handle it on > > > the syslog-ng side. Maybe duplicate the log feed through a separate > > > FIFO and have a script look at that FIFO and build the various RSS > > > feeds according to whatever search criteria your users want to > monitor? > > > > > > -Claus > > > > > > On 7/10/06, Freeman, Michael <[EMAIL PROTECTED]> wrote: > > > > Is there any other way to do it? > > > > > > > > > > > > -----Original Message----- > > > > From: Dukes Clayton [mailto:[EMAIL PROTECTED] > > > > Sent: Monday, July 10, 2006 1:29 PM > > > > To: Freeman, Michael; php-syslog-ng-support@lists.sourceforge.net > > > > Cc: [EMAIL PROTECTED] > > > > Subject: RE: [Php-syslog-ng-support] Log2rss support? > > > > > > > > Be aware that when you use %like% syntax on large amounts of data > > > > it can slow it down a lot :-) > > > > > > > > > > > > -----Original Message----- > > > > From: [EMAIL PROTECTED] > > > > [mailto:[EMAIL PROTECTED] On > > > > Behalf > > > > > > > Of Freeman, Michael > > > > Sent: Monday, July 10, 2006 1:05 PM > > > > To: php-syslog-ng-support@lists.sourceforge.net > > > > Cc: [EMAIL PROTECTED]; Dukes Clayton > > > > Subject: Re: [Php-syslog-ng-support] Log2rss support? > > > > > > > > Here are my initial thoughts on the design: > > > > > > > > I have attached a screenshot of how I think the interface would > > > > look that will allow the user to configure their own particular > RSS feed. > > > > The interface would also allow the user to make the feed public, > > > > so others could subscribe to it. > > > > > > > > The user will be able to build a filter that is essentially going > > > > to be the SQL query used for pulling the data out of mySQL and > > > > generating > > > > > > > the RSS with. > > > > > > > > When a user goes to pull the feed, http://foo.bar/syslog/feed.rss, > > > > > it is really calling a php script that is doing the SQL query and > > > > generating the RSS feed. I'm not sure yet how this will be > > > > possibly, probably just telling apache that .rss is really a .php. > > > > > I'm not sure yet how to build in the authentication system.. Does > > > > anyone have any ideas on that? I think if we could utilize > > > > someones existing session key to validate who they are then we > > > > could also figure out what feed.rss needs to do to build the > > > > actual feed. Users settings for how they want their feeds > > > > displayed would probably also be stored in > > > mySQL. > > > > > > > > -----Original Message----- > > > > From: Dukes Clayton [mailto:[EMAIL PROTECTED] > > > > Sent: Monday, July 10, 2006 12:52 PM > > > > To: Freeman, Michael; [EMAIL PROTECTED] > > > > Cc: php-syslog-ng-support@lists.sourceforge.net > > > > Subject: RE: [Php-syslog-ng-support] Log2rss support? > > > > > > > > I'd have to agree...plus you introduce all the headaches of > > > > maintaining someone elses code and security deficiencies... > > > > > > > > > > > > -----Original Message----- > > > > From: [EMAIL PROTECTED] > > > > [mailto:[EMAIL PROTECTED] On > > > > Behalf > > > > > > > Of Freeman, Michael > > > > Sent: Monday, July 10, 2006 9:32 AM > > > > To: [EMAIL PROTECTED] > > > > Cc: php-syslog-ng-support@lists.sourceforge.net > > > > Subject: Re: [Php-syslog-ng-support] Log2rss support? > > > > > > > > I think that is like trying to kill an ant with a hammer.. I don't > > > > > think people should be forced into using a full blown CMS if all > > > > they want is a front-end to syslog. I'm just trying to fulfill > > > > some requirements from my end-users here at my job and get > > > > something out to > > > them asap. > > > > > > > > -----Original Message----- > > > > From: Jason Taylor [mailto:[EMAIL PROTECTED] > > > > Sent: Friday, July 07, 2006 6:51 PM > > > > To: Freeman, Michael > > > > Cc: [EMAIL PROTECTED]; Claus Lund; > > > > php-syslog-ng-support@lists.sourceforge.net > > > > Subject: RE: [Php-syslog-ng-support] Log2rss support? > > > > > > > > I definitely suggest you look into a CMS (Content Management > > > > System) such as plone, XOOPS, phpNuke, etc. > > > > These various projects already have a well-developed codebase and > > > > probably several modules that can be bent to your needs. > > > > I looked briefly at making php-syslog-ng into a XOOPS module about > > > > > a year and a half ago but got side-tracked. > > > > > > > > The big advantage of using a CMS is that it would really > > > > modularize everything and users would be able to only use the > features they like. > > > > These products will also have a pretty good authentication and > > > > permissioning system which could be useful for limiting which > > > > hosts a user could query or other such niceties. > > > > > > > > Though not rich in time these days, I would be glad to help where > > > > I can in such a project. > > > > > > > > Cheers, > > > > > > > > /Jason > > > > > > > > -- > > > > You can have my Mac when you pry it from my cold, dead hands. > > > > e:[EMAIL PROTECTED] v:514-815-8204 > > > > > > > > <quote who="Freeman, Michael"> > > > > > I'd still be interested in having one centralized location for > > > > > managing all the different elements of syslog-ng. You shouldn't > > > > > necessarily have to go to different places to do this, for > > > > > convenience > > > > > > > > > sake anyways. How it actually gets done I'm not really concerned > > > > > > with yet, this is more of a requirements gathering phase for me. > > > > > > > > > > I have a project at my company to replace CiscoWorks RME with > > > > > Opsware NAS, and the whole reason I am involved in this > > > > > php-syslog-ng stuff now is because NAS didn't have a front-end > > > > > to syslog, like RME provided, and now that I have found this > > > > > tool I have mitigated that requirement, my next goal is to > > > > > extend the syslog analyzer functions to allow people to create > event handlers. > > > > > So it sounds like this functionality is built-in to syslog-ng > > > > > already, I'm interested in wrapping it all into php-syslog-ng, > > > > > if > > > feasible. > > > > > > > > > > So the RSS stuff and the event handlers are not necessarily > > > > intertwined. > > > > > I would separate them so you would have a separate area to build > > > > > > your filters for RSS feeds, and then another area to build your > > > > > event handlers for specific events. > > > > > > > > > > -----Original Message----- > > > > > From: Jason Taylor [mailto:[EMAIL PROTECTED] > > > > > Sent: Friday, July 07, 2006 2:22 PM > > > > > To: Freeman, Michael > > > > > Cc: Claus Lund; php-syslog-ng-support@lists.sourceforge.net > > > > > Subject: Re: [Php-syslog-ng-support] Log2rss support? > > > > > > > > > > > > > > > Something like this might be easier to implement on the back-end > > > > > > syslog-ng side, and not by hitting the database. What about > > > > > tools like > > > > > > > > > swatch or watcher? > > > > > > > > > > How I see it working is creating a second "destination" in the > > > > > syslog-ng config file which sends "pre-filtered" log-output to a > > > > > > file or socket to be u sed as input by one of these log-watcher > > > programs. > > > > > > > > > > When the program sees a hit, it can create the "notify" event > > > > > which could easily be configured as an event to add to an RSS > feed. > > > > > > > > > > If you are dead-set on using the database for this, then you are > > > > > > not > > > > > > > > going to have "real-time" stream processing as in the above > case. > > > > > You > > > > > > > > > will likely be running a periodic query on the database and > > > > > processing > > > > > > > > > that output. > > > > > > > > > > But in either case, what you are after is some kind of alerter > > > > service. > > > > > At present, php-syslog-ng is nothing more than a > > > > > log-presentation service. The alerter stuff can be added on but > > > > > would be entirely separate from the front end. It would be just > > > > > > another consumer of the > > > > > > > > > syslog-ng database. > > > > > > > > > > /Jason > > > > > > > > > > > > > > > -- > > > > > You can have my Mac when you pry it from my cold, dead hands. > > > > > e:[EMAIL PROTECTED] v:514-815-8204 > > > > > > > > > > > > > > > <quote who="Freeman, Michael"> > > > > > > > > > >> Yeah, my idea was to allow the user to build the type of filter > > > > > >> they wanted and underneath the hood it would make a SQL query > > > > >> to the DB grabbing that stuff and generating the RSS. I'm not > > > > >> sure how > > > > > > > >> sophisticated my first attempt would be, but I might make a > > > > >> small release and see what people think and if they find it > > > > >> useful, add some > > > > > > > > > >> of the additional things. The whole reason behind me wanting to > > > > > >> do this is I think while a web front end to syslog is great, it > > > > > >> still requires someone to look at it. I for one use RSS > > > > >> frequently and always have an aggregator application open on > > > > >> one of my screens so it > > > > > > > > >> is easy for me to look over and take notice of things I > > > > >> actually care > > > > > > > > >> about, based on the filter/ruleset I defined for my feed. > > > > >> > > > > >> One could take this concept even further and extend the > > > > >> php-syslog-ng > > > > > > > > >> interface to allow users to create rules and event handlers > > > > >> that would > > > > > > > > > >> be executed by the syslog-ng daemon itself. > > > > >> > > > > >> -----Original Message----- > > > > >> From: Claus Lund [mailto:[EMAIL PROTECTED] > > > > >> Sent: Friday, July 07, 2006 10:43 AM > > > > >> To: Freeman, Michael > > > > >> Subject: Re: [Php-syslog-ng-support] Log2rss support? > > > > >> > > > > >> > > > > >> > > > > >> An RSS feed may be useful in some situations ... but I don't > > > > >> know how > > > > > > > > >> well it would work/useful it would be in very "chatty > > > environments"? > > > > >> Maybe if you can do some filtering on what you subscribe to > > > > >> then it > > > > > > > >> could be used for alerting on certain situations that you want > > > > >> to make > > > > > > > > > >> sure you catch. > > > > >> > > > > >> -Claus > > > > >> > > > > >> > > > > >> > > > > >> On 7/6/06, Freeman, Michael <[EMAIL PROTECTED]> wrote: > > > > >> > > > > >> > > > > >>> Is anyone else besides myself interested in getting RSS feeds > > > > >>> from > > > > > > > >>> php-syslog-ng? I'm thinking about hacking up the log2rss stuff > > > > > >>> to support mySQL and use apache instead of it's own HTTP > > > > >>> daemon in > > > > perl. > > > > >>> > > > > >>> > > > > >>> Netco Government Services has recently acquired Multimax and > > > > >>> is > > > > >>> > > > > >>> > > > > >> changing its name to Multimax Inc. > > > > >>> Visit http://www.multimax.com for more information. > > > > >>> > > > > >>> > > > > >>> > > > > >>> Note: My email address has changed. Please take a moment to > > > > >>> update > > > > >>> > > > > >>> > > > > >> your records with my new address. > > > > >>> > > > > >>> > > > > >>> Using Tomcat but need to do more? Need to support web > > > > >>> services, > > > > >>> > > > > >>> > > > > >> security? > > > > >>> Get stuff done quickly with pre-integrated technology to make > > > > >>> your > > > > > > > >>> job > > > > >>> > > > > >> > > > > >>> easier Download IBM WebSphere Application Server v.1.0.1 based > > > > > >>> on Apache Geronimo > > > > >>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&; > > > > >>> dat= > > > > >>> 12 > > > > >>> 1 > > > > >>> 6 > > > > >>> 42 _______________________________________________ > > > > >>> Php-syslog-ng-support mailing list > > > > >>> Php-syslog-ng-support@lists.sourceforge.net > > > > >>> https://lists.sourceforge.net/lists/listinfo/php-syslog-ng-sup > > > > >>> port > > > > >>> > > > > >>> > > > > >>> > > > > >> > > > > >> Using Tomcat but need to do more? Need to support web services, > > > > >> > > > > > security? > > > > >> Get stuff done quickly with pre-integrated technology to make > > > > >> your job easier Download IBM WebSphere Application Server > > > > >> v.1.0.1 based on > > > > > > > > >> Apache Geronimo > > > > >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&d > > > > >> at=1 > > > > >> 21 > > > > >> 6 > > > > >> 42 _______________________________________________ > > > > >> Php-syslog-ng-support mailing list > > > > >> Php-syslog-ng-support@lists.sourceforge.net > > > > >> https://lists.sourceforge.net/lists/listinfo/php-syslog-ng-supp > > > > >> ort > > > > >> > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------ > > > > ---- > > > > -- > > > > - > > > > Using Tomcat but need to do more? Need to support web services, > > > > security? > > > > Get stuff done quickly with pre-integrated technology to make your > > > > > job > > > > > > > easier Download IBM WebSphere Application Server v.1.0.1 based on > > > > Apache Geronimo > > > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat= > > > > 1216 > > > > 42 _______________________________________________ > > > > Php-syslog-ng-support mailing list > > > > Php-syslog-ng-support@lists.sourceforge.net > > > > https://lists.sourceforge.net/lists/listinfo/php-syslog-ng-support > > > > > > > > > > > > ------------------------------------------------------------------ > > > > ---- > > > > --- Using Tomcat but need to do more? Need to support web > > > > services, security? > > > > Get stuff done quickly with pre-integrated technology to make your > > > > > job > > > > > > > easier Download IBM WebSphere Application Server v.1.0.1 based on > > > > Apache Geronimo > > > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat= > > > > 1216 > > > > 42 _______________________________________________ > > > > Php-syslog-ng-support mailing list > > > > Php-syslog-ng-support@lists.sourceforge.net > > > > https://lists.sourceforge.net/lists/listinfo/php-syslog-ng-support > > > > > > > > > > > > > -------------------------------------------------------------------- > > > ----- Using Tomcat but need to do more? Need to support web > > > services, security? > > > Get stuff done quickly with pre-integrated technology to make your > > > job easier Download IBM WebSphere Application Server v.1.0.1 based > > > on Apache Geronimo > > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=12 > > > 1642 _______________________________________________ > > > Php-syslog-ng-support mailing list > > > Php-syslog-ng-support@lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/php-syslog-ng-support > > > > > > > > > ---------------------------------------------------------------------- > > --- Using Tomcat but need to do more? Need to support web services, > > security? > > Get stuff done quickly with pre-integrated technology to make your job > > > easier Download IBM WebSphere Application Server v.1.0.1 based on > > Apache Geronimo > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=1216 > > 42 _______________________________________________ > > Php-syslog-ng-support mailing list > > Php-syslog-ng-support@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/php-syslog-ng-support > > > > > -- > ======================================= > Clayton Dukes > Network Consulting Engineer, Advanced Services Cisco Systems, Inc. > Office: 919.392.6122 > Fax: : 919.869.1580 > Cell: 813.545.7373 > Email: [EMAIL PROTECTED] > MSN: [EMAIL PROTECTED] > ======================================= > -- ======================================= Clayton Dukes Network Consulting Engineer, Advanced Services Cisco Systems, Inc. Office: 919.392.6122 Fax: : 919.869.1580 Cell: 813.545.7373 Email: [EMAIL PROTECTED] MSN: [EMAIL PROTECTED] ======================================= ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Php-syslog-ng-support mailing list Php-syslog-ng-support@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/php-syslog-ng-support
