On Thu, 17 Oct 2013, masoom alam wrote:
Hi Every one,
I want to ask few questions regarding the design of a system that will
collect logs from client machines and then store them in to our
ElasticSearch storage. Questions are as follows:-
1. It is quite obvious from the Internet and people feedback that
ElasticSearch is a best option for storage of raw logs,
That is a debatable point :-)
personally, I believe that the best storage for raw logs is plain text files.
Now, ES may be the best place to store raw log messages that you want to be able
to do rapid, unplanned searches on. But that is a different use case than just
storage :-)
however, there are
few features that are required: signing of logs is important, so whether we
should use the signing feature of RSYSLOG or elasticsearch (Jetty) since
both provide the feature. From the security point of view we believe that
RSYSLOG signing feature is more important for integrity since RSYSLOG is
placed at the clients end. So whatever log is generated they have a surety
with them that logs are not changed by any other party. Are we thinking in
the right direction? Secondly, if logs are signed by the RSYSLOG how they
are verifiable?
rsyslog's log signing is based on writing the logs to files, not signing each
individual message the way you would need to when putting them into a database,
so use the Elasticsearch mechanism
2. Secondly question is that if we don't want to process the logs other
than a bit formatting and security (Encryption and Signature) as described
above, do we really need to pass them through STORM? Moving logs between
RSYSLOG and ElasticSearch requires logstash with the help of REDIS client.
However, are we following the right path? or there is some design flaw in
that....
I'm neer hears of Storm. Rsyslog can deliver messages directly to Elasticsearch
so there is no technical requirement to use Storm
3. Another question is that ElasticSearch is not a RDBMS. So how to deal
with user accounts i mean access control of various internet users. Do we
need to have a separate instance of ElasticSearch for each client?
that would be a question to ask the ElasticSearch people, and will vary greatly
based on what your use case is.
4. Encryption is another feature that we want to enable for clients for
transporting logs.
rsyslog supports encryption for TCP and RELP logs when transporting them. The
connection to ElasticSearch is via HTTP, I assume that this can be modified to
be HTTPS, but it's possible that rsyslog does not support this (I haven't looked
into this). I would suggest transporting the logs to the ES boxes via rsyslog,
and then have rsyslog deliver the logs to ES via localhost, so that it never
leaves the box. This should eliminate the requirement for the logs to be
encrypted.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
