Both are able to cope with that volume, however we've found RabbitMQ to
have problems with large queues. Kafka holds up better with that.
100GB/day works out to roughly 1MB/sec which is non-trivial for
non-local sources considering WAN bandwidth. It's certainly doable, but
coping with spikes becomes more of a challenge. Kafka has this Zookeeper
dependency which makes WAN tricky, so I'd look at RabbitMQ first. Keep
in mind that RabbitMQ might be more difficult to tune for lots of queue
messages than Kafka.
On 22/03/15 00:37, Pete wrote:
Thanks Kay, I will need to verify the Production version then. I
upgraded the test lab to 1.0.1 last week so I now know what's involved
with that.
One of the remote sites has another 100GB per day or so to be added
(doubling what we do now) so do you think RabbitMQ will still cope
with this?
Happy to put the effort in to use Kafka if that is the preferred
future-proof solution.
Cheers, Pete
On 22 Mar 2015, at 09:15, Kay Röpke <[email protected]
<mailto:[email protected]>> wrote:
If you are truly running 0.93 please do upgrade to 1.0.x asap,
because 0.93 was never really released (it was basically 1.0 alpha).
For that message load both Kafka and AMQP (RabbitMQ works great for
this) are viable. From a complexity perspective RabbitMQ is probably
easier to deploy than Kafka for 4-5 distributed data centers.
Best,
Kay
On 22/03/15 00:11, Pete GS wrote:
Thanks Kay,
The only thing we need to address is unreliable links, which is why
I thought a simple message broker setup at the remote site would be
best.
We currently have four remote sites that will send data back to us
here, three are interstate in Australia and the fourth is in the
USA. We will likely have a fifth site in July which will also be
interstate.
I may not have read the doc's thoroughly enough yet and I may be
missing something obvious, but not having used a message broker
before is probably the main reason I'm struggling to determine the
best solution.
We're happy using NXLog to get the source data, so definitely no
need for Logstash in that case, so it's just the transport that I
need to solve.
As far as Graylog performance goes we have no issues there, we have
two nodes in Production behind an F5 load balancer and are seeing an
average of 3K to 5K messages per second with no issues at all,
although this is still running 0.93 for the moment and I plan on
upgrading to 1.0.1 next week.
I learn best from seeing examples so if there is anything missing in
the docs for me it's probably some example setups integrating Kafka
and NXLog etc. I understand that's probably a bit outside the scope
of the documentation though.
Thanks for the clarifications there, I'll get a Kafka server set up
in the lab during the week and see if I can get my head around it as
that is probably the simplest solution.
Cheers, Pete
On Sunday, 22 March 2015 08:46:56 UTC+10, Kay Röpke wrote:
Hey Pete,
What is it you need to solve exactly?
If you have an unreliable link currently the best (reliable)
solution is a message broker like Kafka or AMQP. The former is a
good choice for high load but less flexibility, while AMQP is
better if you need flexible routing etc.
Logstash/NXLog allow to read from files, which Graylog doesn't
support yet, and pass the data on.
We have some pointers on the docs.graylog.org
<http://docs.graylog.org> site to get started, please let us
know if we are missing details or entire topics. We'll happily
add more content to make it easier to get started.
Radio is deprecated mostly because there are problems around
performance and management. We will most likely move towards a
solution that allows Graylog servers to forward data, because of
the flexibility this provides. You can safely deploy it in the
sense that we will provide detailed migration steps before we
stop supporting it, we just want to discourage its use for
purely performance reasons.
Load balancing TCP traffic across Graylog servers will yield
_much_ better performance than radio has ever provided. In fact
most deployments we support have benefited greatly from moving
to 1.0 and replacing radio with mulitple servers.
We regularly see sustained >100k msg/sec setups with decent
hardware nowadays, something that message queue setups can also
deliver but they usually mean substantially increased complexity.
Disconnected/high latency deployments still benefit from message
broker setups and as such are a viable solution currently.
Feel free to contact me directly if you have more detailed
questions.
Best regards,
Kay
On 21/03/15 23:31, Pete GS wrote:
I see what you're saying there... however is that a little too
complicated adding Logstash into the equation?
I was thinking Servers -> Kafka -> Graylog
Is that possible or am I missing something with Kafka that
would require something like Logstash?
Right now we're struggling to see the need to even add a second
step between the source servers and Graylog but we are being
asked to cater for this by other people in the business, so I
need the simplest and least effort solution.
Cheers, Pete
On Friday, 20 March 2015 18:12:32 UTC+10, Mathieu Grzybek wrote:
Hi,
To collect your logs from the DMZ, we use some RabbitMQ
brokers :
servers === (syslog over UDP or TCP) ===> Logstash ===
(AMQP) ===> RabbitMQ === (AMQP) ===> Logstash === (GELF)
===> Graylog
Mathieu
Le vendredi 20 mars 2015 05:51:11 UTC+1, Pete GS a écrit :
Hi all,
We're looking at adding message sources into our
Graylog setup from a couple of remote sites. There is
the possibility of temporary transit link outages so
sending UDP packets would result in lost messages.
Using TCP will counter this to a certain extent but may
result in messages queuing on sources running NXLog
etc. which we would like to avoid.
I'm think it would be better to set up a message broker
at the remote site and having Graylog subscribe to this
to retrieve the messages instead once the link is
available again.
Does this sound like a sensible idea or is there some
better way to achieve this?
Any recommendations on a preferred message broker etc.?
Any clues on how to get NXLog, Syslog, etc. to log to a
message broker instead of to Graylog directly?
Cheers, Pete
--
You received this message because you are subscribed to the
Google Groups "graylog2" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to [email protected] <javascript:>.
For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google
Groups "graylog2" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to [email protected]
<mailto:[email protected]>.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to a topic in
the Google Groups "graylog2" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/graylog2/yZFX7XkLhTU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
[email protected]
<mailto:[email protected]>.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
