Hi,
As per the jira https://issues.apache.org/jira/browse/KAFKA-1737, I have
set ZKStringSerializer, while creating the ZkClient and still the delete
topic function call is not working:
Version: kafka_2.11-0.8.2.1
*My code looks like this*:
ZkClient zkClient = new ZkClient(localhost:2181, 6000,
Hi,
In the old consumer, I have got just a simple stream of messages, one
by one and if I detected something was wrong, I would destroy my
consumer immediately without commit so once I restart consumer, I will
get the same messages once again because they will be delivered to me
from the last
Hi all,
We're experiencing high cpu usage on some of our development machines when
kafka should be idle. These are single node kafka instances used only for
development, so far we haven't seen any problems on our production system.
It doesn't seem to happen until we deploy a storm topology and
Hi,
In my project Kafka producers won't be in the same network of Kafka brokers
and due to security reasons other ports are blocked.
I would like to know if it is possible to run Kafka brokers on HTTP port
(8080) so that Kafka producers will send Kafka messages over HTTP and
brokers can store
Hi Ewen,
Thank you for your patient explaining. It is very helpful.
Can we assume that the long latency of ProducerPerformance comes from
queuing delay in the buffer and it is related to buffer size?
Thank you!
best,
Yuheng
On Thu, Jul 16, 2015 at 12:21 AM, Ewen Cheslack-Postava
I think the answer here is that the Kafka protocol includes a broker
metadata api. The client uses the broker host(s) you provide to discover
the full list of brokers in the cluster (and the topics+partitions each
manages/leads). The java client has a similar interface via
metadata.brokers.list /
So I found the culprit. On the storm-kafka spout there is a setting:
topology.sleep.spout.wait.strategy.time.ms
this setting defaults to 1ms. Changing it dramatically decreases CPU in
kafka and storm. Does anyone here know anything about this? Should this
be able to peg Kafka's CPU usage?
Maybe what you are looking for is Kafka REST Proxy:
http://docs.confluent.io/1.0/kafka-rest/docs/intro.html
Edward
On Thu, Jul 16, 2015 at 10:24 AM, Chandrashekhar Kotekar
shekhar.kote...@gmail.com wrote:
Hi,
In my project Kafka producers won't be in the same network of Kafka brokers
and
Hey Stefan,
I only see a commit in the failure case. Were you planning to use
auto-commits otherwise? You'd probably want to handle all commits directly
or you'd always be left guessing. But even if you did, I think the main
problem is that your process could fail before a needed commit is sent
Hey Stefan,
I think if you follow the one-commit-per-message approach, you will be
able to achieve exact once semantic. However this would be very expensive
and also everything has to be synchronous in order to make it work.
In that sense, de-duplication on client side seems a more favorable
Hi All,
Is there a way to calculate the amount of memory used per thread in case of
a high level consumer?
I am particularly interested in calculating the memory required by a
process running 10 high level consumer threads for 15 topics with max. file
size set to 100 MB.
Thanks,
Kris
Hi Marina,
Is this reproducible? Can you reproduce this with a smaller number of
events? Did you see any error in broker /server logs?
On Thu, Jul 16, 2015 at 5:42 PM, Marina ppi...@yahoo.com.invalid wrote:
A variation on the posted question - as a new thread so that not to
high-jack the
I think the rough calculation of max memory footprint for each high level
consumer would be:
(Number Of Partitions For All Topics) * fetch.message.max.bytes *
queued.max.message.chunks + (some decompression memory cost for a message)
In your case, it would be 10 times above.
Thanks,
Jiangjie
On Friday 17 July 2015 10:14 AM, Jiangjie Qin wrote:
I think the rough calculation of max memory footprint for each high level
consumer would be:
(Number Of Partitions For All Topics) * fetch.message.max.bytes *
queued.max.message.chunks + (some decompression memory cost for a message)
Is
Thanks a lot Ewen and Edward for your valuable answers. According to new
update from admin side, they can allow TCP only connections on Kafka
brokers.
Now another problem is that we want to keep Kafka brokers in AWS so that
kafka brokers can be auto scaled in/out. As most of instances in AWS do
Yes, those are potential issues if you make your Kafka cluster publicly
accessible. There are some features currently being worked on that could
help address these problems (some security on connections and quotas), but
they are still in progress. You'll probably want a proxy layer to handle
this.
Hello Sivananda,
Calling AdminUtils.deleteTopic just requests topic to be deleted - it does
not actually delete topic immediately. Requests for topic deletion get
saved in ZooKeeper as a node (named by topic name), under
/admin/delete_topics node.
If brokers in the cluster are configured with
Looks like you try to delete a topic that is already in the process of
getting deleted:
NodeExists for /admin/delete_topics/testTopic17
(We can improve the error messages for sure, or maybe even catch the
exception and ignore it)
Gwen
On Thu, Jul 16, 2015 at 3:40 PM, Sivananda Reddy
Chandrashekhar,
If the firewall rules allow any TCP connection on those ports, you can just
use Kafka directly and change the default port. If they actually verify
that its HTTP traffic then you'd have to the REST Proxy Edward mentioned or
another HTTP-based proxy.
-Ewen
On Thu, Jul 16, 2015 at
I'm running into an issue right now where after performing some tests I've
actually exhausted disk storage and am trying to clear it. This is a rather
naive question, but how would i go about manually purging my memory of all
the accumulated messages? I can't find a specific location as it seems
A variation on the posted question - as a new thread so that not to high-jack
the original one...
How long does it usually take to finally delete the topic? I tried to delete
the topic via
/bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic mytopic
and saw that it was marked for
Thanks Foo,
As a follow up, since I was considering that, if I originally published
messages under the topic 'test' with retention set to something like 168
hours, successfully published some amount of messages to that topic (which
are then persisted on disk), and then changed the retention.ms as
If those messages are part of the topic that is specified in the command
line, then yes, they'll get deleted along with all the other messages in
that topic.
On Thu, Jul 16, 2015 at 4:38 PM, JIEFU GONG jg...@berkeley.edu wrote:
Thanks Foo,
As a follow up, since I was considering that, if I
That's a good bet. Running with a few different buffer sizes would verify
this. Adjusting other settings (e.g. max in flight requests, as mentioned
earlier) could also affect latency and throughput.
-Ewen
On Thu, Jul 16, 2015 at 5:20 AM, Yuheng Du yuheng.du.h...@gmail.com wrote:
Hi Ewen,
You can set the retention.ms to something really small, i.e. 60 seconds,
kafka will delete everything older than 60 seconds (eventually).
bin/kafka-topics.sh --zookeeper zookeeper.yoursite.com --alter --topic
your-topic --config retention.ms=6
Remember to set it back to something sane after
25 matches
Mail list logo
