Just checked our version and it's 0.92.3 so I was wrong thankfully :) I'll still be looking to do the 1.0.1 upgrade this week regardless.
I'll look at both RabbitMQ and Kafka and see which one seems to fit better and that I can manage as it sounds like both will do the job we need. Once I've tested it and if we roll it out to Production I'd be happy to share an overview of what's involved if that will help any. Cheers, Pete On Sunday, 22 March 2015 10:23:23 UTC+10, Kay Röpke wrote: > > Both are able to cope with that volume, however we've found RabbitMQ to > have problems with large queues. Kafka holds up better with that. > > 100GB/day works out to roughly 1MB/sec which is non-trivial for non-local > sources considering WAN bandwidth. It's certainly doable, but coping with > spikes becomes more of a challenge. Kafka has this Zookeeper dependency > which makes WAN tricky, so I'd look at RabbitMQ first. Keep in mind that > RabbitMQ might be more difficult to tune for lots of queue messages than > Kafka. > > On 22/03/15 00:37, Pete wrote: > > Thanks Kay, I will need to verify the Production version then. I upgraded > the test lab to 1.0.1 last week so I now know what's involved with that. > > One of the remote sites has another 100GB per day or so to be added > (doubling what we do now) so do you think RabbitMQ will still cope with > this? > > Happy to put the effort in to use Kafka if that is the preferred > future-proof solution. > > Cheers, Pete > > On 22 Mar 2015, at 09:15, Kay Röpke <[email protected] <javascript:>> > wrote: > > If you are truly running 0.93 please do upgrade to 1.0.x asap, because > 0.93 was never really released (it was basically 1.0 alpha). > > For that message load both Kafka and AMQP (RabbitMQ works great for this) > are viable. From a complexity perspective RabbitMQ is probably easier to > deploy than Kafka for 4-5 distributed data centers. > > Best, > Kay > > On 22/03/15 00:11, Pete GS wrote: > > Thanks Kay, > > The only thing we need to address is unreliable links, which is why I > thought a simple message broker setup at the remote site would be best. > > We currently have four remote sites that will send data back to us here, > three are interstate in Australia and the fourth is in the USA. We will > likely have a fifth site in July which will also be interstate. > > I may not have read the doc's thoroughly enough yet and I may be missing > something obvious, but not having used a message broker before is probably > the main reason I'm struggling to determine the best solution. > > We're happy using NXLog to get the source data, so definitely no need > for Logstash in that case, so it's just the transport that I need to solve. > > As far as Graylog performance goes we have no issues there, we have two > nodes in Production behind an F5 load balancer and are seeing an average of > 3K to 5K messages per second with no issues at all, although this is still > running 0.93 for the moment and I plan on upgrading to 1.0.1 next week. > > I learn best from seeing examples so if there is anything missing in the > docs for me it's probably some example setups integrating Kafka and NXLog > etc. I understand that's probably a bit outside the scope of the > documentation though. > > Thanks for the clarifications there, I'll get a Kafka server set up in > the lab during the week and see if I can get my head around it as that is > probably the simplest solution. > > Cheers, Pete > > On Sunday, 22 March 2015 08:46:56 UTC+10, Kay Röpke wrote: >> >> Hey Pete, >> >> What is it you need to solve exactly? >> >> If you have an unreliable link currently the best (reliable) solution is >> a message broker like Kafka or AMQP. The former is a good choice for high >> load but less flexibility, while AMQP is better if you need flexible >> routing etc. >> >> Logstash/NXLog allow to read from files, which Graylog doesn't support >> yet, and pass the data on. >> >> We have some pointers on the docs.graylog.org site to get started, >> please let us know if we are missing details or entire topics. We'll >> happily add more content to make it easier to get started. >> >> Radio is deprecated mostly because there are problems around performance >> and management. We will most likely move towards a solution that allows >> Graylog servers to forward data, because of the flexibility this provides. >> You can safely deploy it in the sense that we will provide detailed >> migration steps before we stop supporting it, we just want to discourage >> its use for purely performance reasons. >> Load balancing TCP traffic across Graylog servers will yield _much_ >> better performance than radio has ever provided. In fact most deployments >> we support have benefited greatly from moving to 1.0 and replacing radio >> with mulitple servers. >> We regularly see sustained >100k msg/sec setups with decent hardware >> nowadays, something that message queue setups can also deliver but they >> usually mean substantially increased complexity. >> >> Disconnected/high latency deployments still benefit from message broker >> setups and as such are a viable solution currently. >> >> Feel free to contact me directly if you have more detailed questions. >> >> Best regards, >> Kay >> >> On 21/03/15 23:31, Pete GS wrote: >> >> I see what you're saying there... however is that a little too >> complicated adding Logstash into the equation? >> >> I was thinking Servers -> Kafka -> Graylog >> >> Is that possible or am I missing something with Kafka that would >> require something like Logstash? >> >> Right now we're struggling to see the need to even add a second step >> between the source servers and Graylog but we are being asked to cater for >> this by other people in the business, so I need the simplest and least >> effort solution. >> >> Cheers, Pete >> >> On Friday, 20 March 2015 18:12:32 UTC+10, Mathieu Grzybek wrote: >>> >>> Hi, >>> >>> To collect your logs from the DMZ, we use some RabbitMQ brokers : >>> servers === (syslog over UDP or TCP) ===> Logstash === (AMQP) ===> >>> RabbitMQ === (AMQP) ===> Logstash === (GELF) ===> Graylog >>> >>> Mathieu >>> >>> Le vendredi 20 mars 2015 05:51:11 UTC+1, Pete GS a écrit : >>>> >>>> Hi all, >>>> >>>> We're looking at adding message sources into our Graylog setup from a >>>> couple of remote sites. There is the possibility of temporary transit link >>>> outages so sending UDP packets would result in lost messages. Using TCP >>>> will counter this to a certain extent but may result in messages queuing >>>> on >>>> sources running NXLog etc. which we would like to avoid. >>>> >>>> I'm think it would be better to set up a message broker at the remote >>>> site and having Graylog subscribe to this to retrieve the messages instead >>>> once the link is available again. >>>> >>>> Does this sound like a sensible idea or is there some better way to >>>> achieve this? >>>> >>>> Any recommendations on a preferred message broker etc.? >>>> >>>> Any clues on how to get NXLog, Syslog, etc. to log to a message >>>> broker instead of to Graylog directly? >>>> >>>> Cheers, Pete >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "graylog2" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> >> >> -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to a topic in the > Google Groups "graylog2" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/graylog2/yZFX7XkLhTU/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected] <javascript:>. > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
