If you are truly running 0.93 please do upgrade to 1.0.x asap, because
0.93 was never really released (it was basically 1.0 alpha).
For that message load both Kafka and AMQP (RabbitMQ works great for
this) are viable. From a complexity perspective RabbitMQ is probably
easier to deploy than Kafka for 4-5 distributed data centers.
Best,
Kay
On 22/03/15 00:11, Pete GS wrote:
Thanks Kay,
The only thing we need to address is unreliable links, which is why I
thought a simple message broker setup at the remote site would be best.
We currently have four remote sites that will send data back to us
here, three are interstate in Australia and the fourth is in the USA.
We will likely have a fifth site in July which will also be interstate.
I may not have read the doc's thoroughly enough yet and I may be
missing something obvious, but not having used a message broker before
is probably the main reason I'm struggling to determine the best solution.
We're happy using NXLog to get the source data, so definitely no need
for Logstash in that case, so it's just the transport that I need to
solve.
As far as Graylog performance goes we have no issues there, we have
two nodes in Production behind an F5 load balancer and are seeing an
average of 3K to 5K messages per second with no issues at all,
although this is still running 0.93 for the moment and I plan on
upgrading to 1.0.1 next week.
I learn best from seeing examples so if there is anything missing in
the docs for me it's probably some example setups integrating Kafka
and NXLog etc. I understand that's probably a bit outside the scope of
the documentation though.
Thanks for the clarifications there, I'll get a Kafka server set up in
the lab during the week and see if I can get my head around it as that
is probably the simplest solution.
Cheers, Pete
On Sunday, 22 March 2015 08:46:56 UTC+10, Kay Röpke wrote:
Hey Pete,
What is it you need to solve exactly?
If you have an unreliable link currently the best (reliable)
solution is a message broker like Kafka or AMQP. The former is a
good choice for high load but less flexibility, while AMQP is
better if you need flexible routing etc.
Logstash/NXLog allow to read from files, which Graylog doesn't
support yet, and pass the data on.
We have some pointers on the docs.graylog.org
<http://docs.graylog.org> site to get started, please let us know
if we are missing details or entire topics. We'll happily add more
content to make it easier to get started.
Radio is deprecated mostly because there are problems around
performance and management. We will most likely move towards a
solution that allows Graylog servers to forward data, because of
the flexibility this provides. You can safely deploy it in the
sense that we will provide detailed migration steps before we stop
supporting it, we just want to discourage its use for purely
performance reasons.
Load balancing TCP traffic across Graylog servers will yield
_much_ better performance than radio has ever provided. In fact
most deployments we support have benefited greatly from moving to
1.0 and replacing radio with mulitple servers.
We regularly see sustained >100k msg/sec setups with decent
hardware nowadays, something that message queue setups can also
deliver but they usually mean substantially increased complexity.
Disconnected/high latency deployments still benefit from message
broker setups and as such are a viable solution currently.
Feel free to contact me directly if you have more detailed questions.
Best regards,
Kay
On 21/03/15 23:31, Pete GS wrote:
I see what you're saying there... however is that a little too
complicated adding Logstash into the equation?
I was thinking Servers -> Kafka -> Graylog
Is that possible or am I missing something with Kafka that would
require something like Logstash?
Right now we're struggling to see the need to even add a second
step between the source servers and Graylog but we are being
asked to cater for this by other people in the business, so I
need the simplest and least effort solution.
Cheers, Pete
On Friday, 20 March 2015 18:12:32 UTC+10, Mathieu Grzybek wrote:
Hi,
To collect your logs from the DMZ, we use some RabbitMQ brokers :
servers === (syslog over UDP or TCP) ===> Logstash === (AMQP)
===> RabbitMQ === (AMQP) ===> Logstash === (GELF) ===> Graylog
Mathieu
Le vendredi 20 mars 2015 05:51:11 UTC+1, Pete GS a écrit :
Hi all,
We're looking at adding message sources into our Graylog
setup from a couple of remote sites. There is the
possibility of temporary transit link outages so sending
UDP packets would result in lost messages. Using TCP will
counter this to a certain extent but may result in
messages queuing on sources running NXLog etc. which we
would like to avoid.
I'm think it would be better to set up a message broker
at the remote site and having Graylog subscribe to this
to retrieve the messages instead once the link is
available again.
Does this sound like a sensible idea or is there some
better way to achieve this?
Any recommendations on a preferred message broker etc.?
Any clues on how to get NXLog, Syslog, etc. to log to a
message broker instead of to Graylog directly?
Cheers, Pete
--
You received this message because you are subscribed to the
Google Groups "graylog2" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to [email protected] <javascript:>.
For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google
Groups "graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
