I am using sarama golang kafka 1.8.1 client (
https://github.com/Shopify/sarama) to send messages to message queue once
in 3secs and this task drives cpu consumption to 130% on the quad-cpu
blade; The number stays this high regardless of number of
partitions/consumers According to results of
# I looked the documents of kafka and I see that there is no way a
consume instance can
read specific messages from partition.
With Kafka you read messages from the beginning multiple times. Since you
say later that
you do not have many messages per topic, you can iterate over the
Hi Victor, two points:
- Based on the backtrace, you are using a very old version of Sarama. You
might have better luck using a more recent stable version.
- Are you setting `MaxBufferTime` in the configuration to 0 or a very small
value? If so the loop will spin on that timer. Try making this
I think of it as default.offset :)
Realistically though, changing the configuration name will probably
cause more confusion than leaving a bad name as is.
On Tue, Apr 14, 2015 at 11:10 AM, James Cheng jch...@tivo.com wrote:
What to do when there is no initial offset in ZooKeeper or if an offset
I set it to 10ms and it did fix cpu consumption problem but i don't see any
messages coming up in consumer queue
On Wed, Apr 15, 2015 at 1:57 PM, Evan Huus evan.h...@shopify.com wrote:
The field is a time.Duration (https://golang.org/pkg/time/#Duration) and
it
is the maximum duration to
On Wed, Apr 15, 2015 at 1:33 PM, Victor L vlyamt...@gmail.com wrote:
Hi Evan,
It's hardcoded:
MaxBufferTime = 1000
Without any units this defaults to nanoseconds, which means your timer is
spinning every microsecond. You probably mean 1000 * time.Millisecond?
MaxBufferedBytes=
On 04/15/15 09:31, Manoj Khangaonkar wrote:
# I looked the documents of kafka and I see that there is no way a
consume instance can
read specific messages from partition.
With Kafka you read messages from the beginning multiple times. Since you
say later that
you do not have
I see the following error in consumer;
Unable to Receive Message:kafka server: The requested offset is outside the
range of offsets maintained by the server for the given topic/partition
On Wed, Apr 15, 2015 at 2:09 PM, Victor L vlyamt...@gmail.com wrote:
I set it to 10ms and it did fix cpu
Check the offset you're asking for, and check what offsets actually exist
on your cluster. It sounds like they don't line up.
On Wed, Apr 15, 2015 at 2:10 PM, Victor L vlyamt...@gmail.com wrote:
I see the following error in consumer;
Unable to Receive Message:kafka server: The requested offset
Hi Evan,
It's hardcoded:
MaxBufferTime = 1000
MaxBufferedBytes= 1024
How's it versioned? Should i just download zip file from project github?
On Wed, Apr 15, 2015 at 12:48 PM, Evan Huus evan.h...@shopify.com wrote:
Hi Victor, two points:
- Based on the backtrace, you are using a
I inherited this code, it just set to 1000, what would be reasonable time:
1000 * time.Millisecond?
On Wed, Apr 15, 2015 at 1:41 PM, Evan Huus evan.h...@shopify.com wrote:
On Wed, Apr 15, 2015 at 1:33 PM, Victor L vlyamt...@gmail.com wrote:
Hi Evan,
It's hardcoded:
MaxBufferTime =
The field is a time.Duration (https://golang.org/pkg/time/#Duration) and it
is the maximum duration to buffer messages before triggering a flush to the
broker. The default was 1 millisecond (this value does not even exist in
the most recent release). Setting it to higher values trades off latency
I meant you can read messages multiple times if you want to.
Yes, you would store offset and request reading from an offset with Simple
Consumer API to implement once and only once delivery.
regards
On Wed, Apr 15, 2015 at 10:55 AM, Pete Wright pwri...@rubiconproject.com
wrote:
On 04/15/15
i have 0.8.1 producer/consumer... I start producer first but when i start
consumer i see the following :
Apr 15 21:26:29 node-1cc1de6d1224 docker[32360]:
INFO|10cf22c161e3|1|consumer|main.go:34|2015/04/15 21:26:29|Starting
ConsumerService
Apr 15 21:26:29 node-1cc1de6d1224 docker[32360]:
Hello Tao,
Do you think the solution to KAFKA-2056 will resolve this issue? It will be
included in 0.8.3 release.
Guozhang
On Wed, Apr 15, 2015 at 2:21 PM, tao xiao xiaotao...@gmail.com wrote:
Hi team,
I discovered an issue that when a high level consumer with roundrobin
assignment
Guozhang,
No, I don't think the patch of KAFKA-2056 would fix this problem. The NPE
is thrown at the line that is called before the fix executes. But I do
notice that the code in trunk did fix the issue by ensuring the size of map
returned from ctx.consumersForTopic is 0. So the code in trunk is
the one in zookeeper is 123 and consumer is asking for 67375... Don't
understand where's the second value coming from
On Wed, Apr 15, 2015 at 2:17 PM, Evan Huus evan.h...@shopify.com wrote:
Check the offset you're asking for, and check what offsets actually exist
on your cluster. It sounds
Hi team,
I discovered an issue that when a high level consumer with roundrobin
assignment strategy consumes a topic that hasn't been created on broker a
NPE exception is thrown during partition rebalancing phase. I use Kafka
0.8.2.1
Here is the step to reproduce:
1. create a high level consumer
Hi All,
What is the lightest way to check if the connection to Kafka is up or not?
Currently I'm having the producer attempt to send a message to a topic, and
then catching a TimeoutException and handling logic inside of the catch
statement. This doesn't seem like the right way to do this,
19 matches
Mail list logo
