Hi, I'd like to add some bits of info here and there, too. I usually react to the "rsyslog+Elasticsearch" keyword combination :p
2 - I think the fastest and most scalable way at the moment is to use the rsyslog to Elasticsearch output plugin: omelasticsearch<http://www.rsyslog.com/doc/omelasticsearch.html>. I like David's idea of using TLS syslog and putting a rsyslog on the same machine as ES and do the HTTP requests locally. For searching, if you want to use Kibana<http://www.elasticsearch.org/overview/kibana/>I think you can use something like Apache as a reverse proxy to do the encryption and authentication part. They provide a sample Apache conf that might help: https://github.com/elasticsearch/kibana/blob/master/sample/apache_ldap.conf If you need help in setting up rsyslog to output logs in a Kibana-friendly way, check this blog post I wrote a while ago: http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/ 3 - to do this, you need some external mechanism to control access. If you do the Apache LDAP thing mentioned above, it should work to make one index per user, and then limit the access of each user to her own index by link (e.g.: http://host:port/*index-of-this-user*/) Alternatively, you might want to check out the ES jetty plugin, for securing and restricting access: https://github.com/sonian/elasticsearch-jetty 2013/10/17 David Lang <[email protected]> > On Thu, 17 Oct 2013, masoom alam wrote: > > Hi Every one, >> >> I want to ask few questions regarding the design of a system that will >> collect logs from client machines and then store them in to our >> ElasticSearch storage. Questions are as follows:- >> >> >> 1. It is quite obvious from the Internet and people feedback that >> ElasticSearch is a best option for storage of raw logs, >> > > That is a debatable point :-) > > personally, I believe that the best storage for raw logs is plain text > files. > > Now, ES may be the best place to store raw log messages that you want to > be able to do rapid, unplanned searches on. But that is a different use > case than just storage :-) > > > however, there are >> few features that are required: signing of logs is important, so >> whether we >> should use the signing feature of RSYSLOG or elasticsearch (Jetty) since >> both provide the feature. From the security point of view we believe >> that >> RSYSLOG signing feature is more important for integrity since RSYSLOG is >> placed at the clients end. So whatever log is generated they have a >> surety >> with them that logs are not changed by any other party. Are we thinking >> in >> the right direction? Secondly, if logs are signed by the RSYSLOG how >> they >> are verifiable? >> > > rsyslog's log signing is based on writing the logs to files, not signing > each individual message the way you would need to when putting them into a > database, so use the Elasticsearch mechanism > > > 2. Secondly question is that if we don't want to process the logs other >> than a bit formatting and security (Encryption and Signature) as >> described >> above, do we really need to pass them through STORM? Moving logs between >> RSYSLOG and ElasticSearch requires logstash with the help of REDIS >> client. >> However, are we following the right path? or there is some design flaw >> in >> that.... >> > > I'm neer hears of Storm. Rsyslog can deliver messages directly to > Elasticsearch so there is no technical requirement to use Storm > > > 3. Another question is that ElasticSearch is not a RDBMS. So how to deal >> with user accounts i mean access control of various internet users. Do >> we >> need to have a separate instance of ElasticSearch for each client? >> > > that would be a question to ask the ElasticSearch people, and will vary > greatly based on what your use case is. > > > 4. Encryption is another feature that we want to enable for clients for >> transporting logs. >> > > rsyslog supports encryption for TCP and RELP logs when transporting them. > The connection to ElasticSearch is via HTTP, I assume that this can be > modified to be HTTPS, but it's possible that rsyslog does not support this > (I haven't looked into this). I would suggest transporting the logs to the > ES boxes via rsyslog, and then have rsyslog deliver the logs to ES via > localhost, so that it never leaves the box. This should eliminate the > requirement for the logs to be encrypted. > > David Lang > > ______________________________**_________________ > rsyslog mailing list > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> > http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
