Hi Radu and David, Thanks for the suggestion regarding transporting logs between RSYSLOG and RSYSLOG and then storage in to ElasticSearch (ES). However, what about signing feature, does jetty provides signing of individual messages that are going to be stored in ES?
Secondly, do we need to write a custom module for Kibana for visualizing the verification of signatures for individual messages. To give an example, consider a client that may want to verify that whether a particular log message was generated in its premise and it is unaltered or not through Kibana? In this regard what about PKI infrastructure required. I mean private keys should reside at the client side for signing messages. The ambiguity for me in this scenario is that ES is at our side. If signature means hash only, how client can verify that log message was not altered at our side? Kindly guide. On Fri, Oct 18, 2013 at 8:20 AM, masoom alam <[email protected]> wrote: > Thanks for your kind suggestions. > > I will get back to you after finalizing these details. > > Thanks. > > > On Thu, Oct 17, 2013 at 12:43 PM, Radu Gheorghe > <[email protected]>wrote: > >> Hi, >> >> I'd like to add some bits of info here and there, too. I usually react to >> the "rsyslog+Elasticsearch" keyword combination :p >> >> 2 - I think the fastest and most scalable way at the moment is to use the >> rsyslog to Elasticsearch output plugin: >> omelasticsearch<http://www.rsyslog.com/doc/omelasticsearch.html>. >> I like David's idea of using TLS syslog and putting a rsyslog on the same >> machine as ES and do the HTTP requests locally. >> >> For searching, if you want to use >> Kibana<http://www.elasticsearch.org/overview/kibana/>I think you can >> use something like Apache as a reverse proxy to do the >> encryption and authentication part. They provide a sample Apache conf that >> might help: >> >> https://github.com/elasticsearch/kibana/blob/master/sample/apache_ldap.conf >> >> If you need help in setting up rsyslog to output logs in a Kibana-friendly >> way, check this blog post I wrote a while ago: >> http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/ >> >> 3 - to do this, you need some external mechanism to control access. If you >> do the Apache LDAP thing mentioned above, it should work to make one index >> per user, and then limit the access of each user to her own index by link >> (e.g.: http://host:port/*index-of-this-user*/) >> >> Alternatively, you might want to check out the ES jetty plugin, for >> securing and restricting access: >> https://github.com/sonian/elasticsearch-jetty >> >> >> >> 2013/10/17 David Lang <[email protected]> >> >> > On Thu, 17 Oct 2013, masoom alam wrote: >> > >> > Hi Every one, >> >> >> >> I want to ask few questions regarding the design of a system that will >> >> collect logs from client machines and then store them in to our >> >> ElasticSearch storage. Questions are as follows:- >> >> >> >> >> >> 1. It is quite obvious from the Internet and people feedback that >> >> ElasticSearch is a best option for storage of raw logs, >> >> >> > >> > That is a debatable point :-) >> > >> > personally, I believe that the best storage for raw logs is plain text >> > files. >> > >> > Now, ES may be the best place to store raw log messages that you want to >> > be able to do rapid, unplanned searches on. But that is a different use >> > case than just storage :-) >> > >> > >> > however, there are >> >> few features that are required: signing of logs is important, so >> >> whether we >> >> should use the signing feature of RSYSLOG or elasticsearch (Jetty) >> since >> >> both provide the feature. From the security point of view we believe >> >> that >> >> RSYSLOG signing feature is more important for integrity since >> RSYSLOG is >> >> placed at the clients end. So whatever log is generated they have a >> >> surety >> >> with them that logs are not changed by any other party. Are we >> thinking >> >> in >> >> the right direction? Secondly, if logs are signed by the RSYSLOG how >> >> they >> >> are verifiable? >> >> >> > >> > rsyslog's log signing is based on writing the logs to files, not signing >> > each individual message the way you would need to when putting them >> into a >> > database, so use the Elasticsearch mechanism >> > >> > >> > 2. Secondly question is that if we don't want to process the logs >> other >> >> than a bit formatting and security (Encryption and Signature) as >> >> described >> >> above, do we really need to pass them through STORM? Moving logs >> between >> >> RSYSLOG and ElasticSearch requires logstash with the help of REDIS >> >> client. >> >> However, are we following the right path? or there is some design >> flaw >> >> in >> >> that.... >> >> >> > >> > I'm neer hears of Storm. Rsyslog can deliver messages directly to >> > Elasticsearch so there is no technical requirement to use Storm >> > >> > >> > 3. Another question is that ElasticSearch is not a RDBMS. So how to >> deal >> >> with user accounts i mean access control of various internet users. >> Do >> >> we >> >> need to have a separate instance of ElasticSearch for each client? >> >> >> > >> > that would be a question to ask the ElasticSearch people, and will vary >> > greatly based on what your use case is. >> > >> > >> > 4. Encryption is another feature that we want to enable for clients >> for >> >> transporting logs. >> >> >> > >> > rsyslog supports encryption for TCP and RELP logs when transporting >> them. >> > The connection to ElasticSearch is via HTTP, I assume that this can be >> > modified to be HTTPS, but it's possible that rsyslog does not support >> this >> > (I haven't looked into this). I would suggest transporting the logs to >> the >> > ES boxes via rsyslog, and then have rsyslog deliver the logs to ES via >> > localhost, so that it never leaves the box. This should eliminate the >> > requirement for the logs to be encrypted. >> > >> > David Lang >> > >> > ______________________________**_________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/**mailman/listinfo/rsyslog< >> http://lists.adiscon.net/mailman/listinfo/rsyslog> >> > http://www.rsyslog.com/**professional-services/< >> http://www.rsyslog.com/professional-services/> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > DON'T LIKE THAT. >> > >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
