Re: Kafka Support as Service

Hi Diego, Confluent offers support for Apache Kafka. https://www.confluent.io/ Cheers, Roger On Wed, Apr 12, 2017 at 11:14 AM, Diego Paes Ramalho Pereira < diego.pere...@b3.com.br> wrote: > Hello, > > > > I work for a Stock Exchange in Brazil and We are looking for a company > that can

Re: Role of Zookeeper in Kafka Rest Proxy

Yes On Fri, Feb 17, 2017 at 10:06 PM, Jeff Widman wrote: > Will this new release use a new consumer? > > On Feb 16, 2017 11:33 PM, wrote: > > > You can't integrate 3.1.1 REST Proxy with a secure cluster because it > uses > > the old consumer API (hence

Re: [DISCUSS] KIP-118: Drop Support for Java 7 in Kafka 0.11

This is great. Thanks, Ismael. On Fri, Feb 3, 2017 at 7:35 AM, Grant Henke wrote: > Looks good to me. Thanks for handling the KIP. > > On Fri, Feb 3, 2017 at 8:49 AM, Damian Guy wrote: > > > Thanks Ismael. Makes sense to me. > > > > On Fri, 3 Feb

Re: Messages corrupted in kafka

Are you using snappy compression? There was a bug with snappy that caused corrupt messages. Sent from my iPhone > On Mar 29, 2016, at 8:15 AM, sunil kalva wrote: > > Hi > Do we store message crc also on disk, and server verifies same when we are > reading messages back

Re: open source projects based on kafka

Hi Li, You might take a look at Apache Samza. It's conceptually simple but powerful and makes heavy use of Kafka. Best, Roger Sent from my iPhone > On Sep 12, 2015, at 10:34 PM, Li Tao wrote: > > Hi Hackers, > > This is Lee, a learner of kafka, i have read the

Re: [ANNOUNCE] Burrow - Consumer Lag Monitoring as a Service

Issue is here: https://github.com/linkedin/Burrow/issues/3 On Fri, Jun 12, 2015 at 11:34 AM, Roger Hoover roger.hoo...@gmail.com wrote: Will do. Thanks Sent from my iPhone On Jun 12, 2015, at 10:43 AM, Todd Palino tpal...@gmail.com wrote: Can you open an issue on the github page

Re: [ANNOUNCE] Burrow - Consumer Lag Monitoring as a Service

Hi, I was trying to give burrow a try and got a ZK error invalid ACL specified. Any suggestions on what's going wrong? 1434044348908673512 [Critical] Cannot get ZK notifier lock: zk: invalid ACL specified Here's my config: [general] logdir=log logconfig=logging.cfg pidfile=burrow.pid

Re: [ANNOUNCE] Burrow - Consumer Lag Monitoring as a Service

you set up ACLs within it? I'm not able to see this on our ZK (3.4.6 with no ACLs). -Todd On Fri, Jun 12, 2015 at 9:34 AM, Roger Hoover roger.hoo...@gmail.com wrote: Hi, I was trying to give burrow a try and got a ZK error invalid ACL specified. Any suggestions on what's going wrong

Re: [ANNOUNCE] Burrow - Consumer Lag Monitoring as a Service

Will do. Thanks Sent from my iPhone On Jun 12, 2015, at 10:43 AM, Todd Palino tpal...@gmail.com wrote: Can you open an issue on the github page please, and we can investigate further there? -Todd On Fri, Jun 12, 2015 at 10:22 AM, Roger Hoover roger.hoo...@gmail.com wrote: Thanks

Re: kafka.message.InvalidMessageException: Message is corrupt crc

Are you using snappy compression? I ran into an issue with message corruption with the new producer, snappy compression, and broker restart. On Mon, May 4, 2015 at 12:55 AM, scguo sc...@iflytek.com wrote: Hi Here is my questions. kafka.message.InvalidMessageException: Message is

Re: Message corruption with new Java client + snappy + broker restart

Oops. I originally sent this to the dev list but meant to send it here. Hi, When using Samza 0.9.0 which uses the new Java producer client and snappy enabled, I see messages getting corrupted on the client side. It never happens with the old producer and it never happens with lz4, gzip, or

Re: Support for Java 1.8?

Resurrecting an old thread. Are people running Kafka on Java 8 now? On Sun, Aug 10, 2014 at 11:44 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Just curious if you saw any issues with Java 1.8 or if everything went smoothly? Otis -- Performance Monitoring * Log Analytics *

Re: Support for Java 1.8?

-XX:ErrorFile=logs/hs_err.log -Jon On Mar 17, 2015, at 10:26 AM, Roger Hoover roger.hoo...@gmail.com wrote: Resurrecting an old thread. Are people running Kafka on Java 8 now? On Sun, Aug 10, 2014 at 11:44 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Just curious if you saw

Re: Database Replication Question

Hi Jonathan, TCP will take care of re-ordering the packets. On Wed, Mar 4, 2015 at 6:05 PM, Jonathan Hodges hodg...@gmail.com wrote: Thanks James. This is really helpful. Another extreme edge case might be that the single producer is sending the database log changes and the network causes

Re: Increasing the throughput of Kafka Publisher

is their at Logstash end and perhaps we need to look for an alternative to the same. Do let me know your observation and understanding as well. Thanks! On Thu, Mar 5, 2015 at 1:13 PM, Roger Hoover roger.hoo...@gmail.com wrote: Seeing around 5k msgs/s. The messages are small (average 42 bytes after

Re: Increasing the throughput of Kafka Publisher

what's the throughput you have reaching. Thanks! On Thu, Mar 5, 2015 at 12:56 PM, Roger Hoover roger.hoo...@gmail.com wrote: Hi Vineet, Try enabling compression. That improves throughput 3-4x usually for me. Also, you can use async mode if you're willing to trade some chance of dropping

Re: Increasing the throughput of Kafka Publisher

Hi Vineet, Try enabling compression. That improves throughput 3-4x usually for me. Also, you can use async mode if you're willing to trade some chance of dropping messages for more throughput. kafka { codec = 'json' broker_list = localhost:9092 topic_id = blah

Re: [DISCUSSION] adding the serializer api back to the new java producer

It also makes it possible to do validation on the server side or make other tools that inspect or display messages (e.g. the various command line tools) and do this in an easily pluggable way across tools. I agree that it's valuable to have a standard way to plugin serialization across many

Re: Aeron, a high throughput, low latency messaging system

Hi Daniel, Thanks for sharing this. Looks like a great project. I probably don't know enough to give a great answer but will throw in my 2c anyway. I think Kafka prioritizes throughput and aeron prioritizes latency. As you mentioned, maybe Aeron could replace the current Kafka TCP protocol.

Re: Cannot connect to Kafka from outside of EC2

Just a guess but could it be a firewall issue? Did you enable connections to port 9092 from outside EC2 in a security group? Can you telnet to each broker IP and port? On Tue, Oct 28, 2014 at 10:01 AM, Sameer Yami sy...@ailive.net wrote: There was a typo earlier. This is the output -

Re: Live Upgrade

At least two including the leader? On Fri, Oct 17, 2014 at 8:12 AM, Guozhang Wang wangg...@gmail.com wrote: Hi Balaji, You could do a rolling bounce of the brokers to do the in-place upgrade if your partitions have at least two replicas. After that you may probably need to rebalance the

Re: Kafka/Zookeeper deployment Questions

Casey, Could you describe a little more about how these would help manage a cluster? My understanding is that Consul provides service discovery and leader election. Kafka already uses ZooKeeper for brokers to discover each other and elect partition leaders. Kafka high-level consumers use ZK to

Re: Kafka/Zookeeper deployment Questions

what works and what doesn't. ___ From: Roger Hoover [roger.hoo...@gmail.com] Sent: Friday, October 17, 2014 12:26 PM To: users@kafka.apache.org Subject: Re: Kafka/Zookeeper deployment Questions Casey, Could you describe a little more about how these would

Re: Command line tool for topic metadata?

ephemeralOwner = 0x0 dataLength = 51 numChildren = 0 On Tue, Oct 14, 2014 at 11:58 AM, Roger Hoover roger.hoo...@gmail.com wrote: Hi, How do I check if a topic is configured for compaction? Is there a command-line tool to see topic metadata like cleanup.policy=compact? Thanks, Roger

Re: Command line tool for topic metadata?

and the broker defaults for leader? Thanks, Roger On Tue, Oct 14, 2014 at 12:17 PM, Roger Hoover roger.hoo...@gmail.com wrote: Oh, duh. I see it in the kafka-topics tool as well. Sorry for the distraction. kafka-topics.sh --zookeeper localhost:2181 --describe --topic foo Topic:foo PartitionCount:1

Re: Command line tool for topic metadata?

defaults. So using the kafka-topics tool should suffice. On Tue, Oct 14, 2014 at 1:32 PM, Roger Hoover roger.hoo...@gmail.com wrote: I still have a question though. Is there a definitive way to tell if a topic is configured for compaction? The way it seems to work now is that the ZK config

Re: [Issue] Cannot connect to Kafka cluster on GCE from local machine

I wouldn't say that Kafka's making it difficult. The cloud environment is making it difficult. The VM that the Kafka broker is running on can only see it's private IP (at the OS level) so you have to add the advertised.host.name config so that it knows what public IP is assigned to it. On Fri,

Re: [Issue] Cannot connect to Kafka cluster on GCE from local machine

regards, James Roger Hoover roger.hoo...@gmail.com 於 2014/6/14 上午12:32 寫道： I wouldn't say that Kafka's making it difficult. The cloud environment is making it difficult. The VM that the Kafka broker is running on can only see it's private IP (at the OS level) so you have to add

Re: [Issue] Cannot connect to Kafka cluster on GCE from local machine

Actually, there wasn't a way to do it prior to 0.8.1 On Fri, Jun 13, 2014 at 3:30 PM, Roger Hoover roger.hoo...@gmail.com wrote: Yes, I believe that prior to Kafka 0.8 there was no easy way for external clients to talk to Kafka brokers running in a cloud environment. I wrote a blog post

Re: How produce to remote Kafka Cluster over ssh tunneling

I think setting these is not a good idea b/c only apply to the specific client where you've setup the tunnel. Other clients cannot use these settings advertised.host.name=localhost advertised.port=19092 You probably need to figure out another way such as 1) Setting up a local mapping on your

Re: What's new in Kafka 0.8.1?

Thanks, Jay. Great write up. I noticed a bad link to the docs for basic operations ( http://localhost/documentation.html#basic_ops). It's in the paragraph that starts with We also improved a lot of operational activities Roger On Wed, Mar 12, 2014 at 8:57 PM, Jay Kreps

Re: New Producer Public API

reactive libs and things like J8's ComposableFuture. Thanks again, C On Fri, Jan 24, 2014 at 2:46 PM, Roger Hoover roger.hoo...@gmail.com wrote: A couple comments: 1) Why does the config use

Re: Questions about producer API

look making the zookeeper config change on the producer soon. But that is something to discuss on a JIRA. Thanks, Neha On Wed, Oct 30, 2013 at 9:53 AM, Roger Hoover roger.hoo...@gmail.com wrote: Hi, I'm still getting started with Kafka and was curious why there is an asymmetry

Re: Why Apache Kafka is bettern than any other Message System?

I'll give you my take in case it helps. Kafka achieves great throughput + durability because 1) The broker does minimal work a) Routing is done by producers b) State management is done by consumers Other messaging brokers typically have to keep track of which messages have been

Re: How to run a single unit test with ./sbt

it isn't very helpful in saying exactly where the test failed; my environment is probably messed up but I know of one or two others who are having similar issues. Joel On Fri, Oct 25, 2013 at 4:24 PM, Roger Hoover roger.hoo...@gmail.com wrote: Hi, I'm new to Scala but working on a simple

Re: Broker bind address versus published hostname in ZooKeeper

/KAFKA-1092 Please take a look and let me know if anything else is needed. Cheers, Roger On Fri, Oct 25, 2013 at 4:05 PM, Roger Hoover roger.hoo...@gmail.comwrote: Ok. I'm working on it. On Thu, Oct 24, 2013 at 10:02 AM, Timothy Chen tnac...@gmail.com wrote: Hi Folks/Roger, Unfortunately I

How to run a single unit test with ./sbt

Hi, I'm new to Scala but working on a simple patch for a configuration change and want to run just my unit tests. When I run ./sbt test-only, it executes all sorts of other tests but not the one I want. Is there an easy way to run a single test? Any help is appreciated. $ ./sbt test-only

Re: Broker bind address versus published hostname in ZooKeeper

. I also created zkHost.port since the internal and external ports that's exposed might be different as well. Tim On Thu, Oct 17, 2013 at 3:13 PM, Roger Hoover roger.hoo...@gmail.com wrote: Hi all, I'm getting started experimenting with Kafka and ran into a configuration

Broker bind address versus published hostname in ZooKeeper

Hi all, I'm getting started experimenting with Kafka and ran into a configuration issue. Currently, in server.properties, you can configure host.name which gets used for two purposes: 1) to bind the socket 2) to publish the broker details to ZK for clients to use. There are times when these two

Re: Added an operations page for 0.8 to site

Thank you, Jay. When talking about flush rates, I think you mean the opposite of what was said here: However very high application flush rates can lead to high latency when the flush does occur. should be However very low application flush rates (infrequent flushes) can lead to high latency