Thanks for your kind suggestions. I will get back to you after finalizing these details.
Thanks. On Thu, Oct 17, 2013 at 12:43 PM, Radu Gheorghe <[email protected]>wrote: > Hi, > > I'd like to add some bits of info here and there, too. I usually react to > the "rsyslog+Elasticsearch" keyword combination :p > > 2 - I think the fastest and most scalable way at the moment is to use the > rsyslog to Elasticsearch output plugin: > omelasticsearch<http://www.rsyslog.com/doc/omelasticsearch.html>. > I like David's idea of using TLS syslog and putting a rsyslog on the same > machine as ES and do the HTTP requests locally. > > For searching, if you want to use > Kibana<http://www.elasticsearch.org/overview/kibana/>I think you can > use something like Apache as a reverse proxy to do the > encryption and authentication part. They provide a sample Apache conf that > might help: > https://github.com/elasticsearch/kibana/blob/master/sample/apache_ldap.conf > > If you need help in setting up rsyslog to output logs in a Kibana-friendly > way, check this blog post I wrote a while ago: > http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/ > > 3 - to do this, you need some external mechanism to control access. If you > do the Apache LDAP thing mentioned above, it should work to make one index > per user, and then limit the access of each user to her own index by link > (e.g.: http://host:port/*index-of-this-user*/) > > Alternatively, you might want to check out the ES jetty plugin, for > securing and restricting access: > https://github.com/sonian/elasticsearch-jetty > > > > 2013/10/17 David Lang <[email protected]> > > > On Thu, 17 Oct 2013, masoom alam wrote: > > > > Hi Every one, > >> > >> I want to ask few questions regarding the design of a system that will > >> collect logs from client machines and then store them in to our > >> ElasticSearch storage. Questions are as follows:- > >> > >> > >> 1. It is quite obvious from the Internet and people feedback that > >> ElasticSearch is a best option for storage of raw logs, > >> > > > > That is a debatable point :-) > > > > personally, I believe that the best storage for raw logs is plain text > > files. > > > > Now, ES may be the best place to store raw log messages that you want to > > be able to do rapid, unplanned searches on. But that is a different use > > case than just storage :-) > > > > > > however, there are > >> few features that are required: signing of logs is important, so > >> whether we > >> should use the signing feature of RSYSLOG or elasticsearch (Jetty) > since > >> both provide the feature. From the security point of view we believe > >> that > >> RSYSLOG signing feature is more important for integrity since RSYSLOG > is > >> placed at the clients end. So whatever log is generated they have a > >> surety > >> with them that logs are not changed by any other party. Are we > thinking > >> in > >> the right direction? Secondly, if logs are signed by the RSYSLOG how > >> they > >> are verifiable? > >> > > > > rsyslog's log signing is based on writing the logs to files, not signing > > each individual message the way you would need to when putting them into > a > > database, so use the Elasticsearch mechanism > > > > > > 2. Secondly question is that if we don't want to process the logs > other > >> than a bit formatting and security (Encryption and Signature) as > >> described > >> above, do we really need to pass them through STORM? Moving logs > between > >> RSYSLOG and ElasticSearch requires logstash with the help of REDIS > >> client. > >> However, are we following the right path? or there is some design flaw > >> in > >> that.... > >> > > > > I'm neer hears of Storm. Rsyslog can deliver messages directly to > > Elasticsearch so there is no technical requirement to use Storm > > > > > > 3. Another question is that ElasticSearch is not a RDBMS. So how to > deal > >> with user accounts i mean access control of various internet users. Do > >> we > >> need to have a separate instance of ElasticSearch for each client? > >> > > > > that would be a question to ask the ElasticSearch people, and will vary > > greatly based on what your use case is. > > > > > > 4. Encryption is another feature that we want to enable for clients > for > >> transporting logs. > >> > > > > rsyslog supports encryption for TCP and RELP logs when transporting them. > > The connection to ElasticSearch is via HTTP, I assume that this can be > > modified to be HTTPS, but it's possible that rsyslog does not support > this > > (I haven't looked into this). I would suggest transporting the logs to > the > > ES boxes via rsyslog, and then have rsyslog deliver the logs to ES via > > localhost, so that it never leaves the box. This should eliminate the > > requirement for the logs to be encrypted. > > > > David Lang > > > > ______________________________**_________________ > > rsyslog mailing list > > http://lists.adiscon.net/**mailman/listinfo/rsyslog< > http://lists.adiscon.net/mailman/listinfo/rsyslog> > > http://www.rsyslog.com/**professional-services/< > http://www.rsyslog.com/professional-services/> > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
