Hi Hemanth, it was introduced on 04/04/2015 at github, so after the 0.8.2.0
version, it will be part of the 0.9.0.0 release.
On Thu, Nov 12, 2015 at 8:39 AM Hemanth Yamijala wrote:
> Thanks, Ewen. I did know passing 0 would work, but didn't find an API to
> get back the bound port. Which version
I don't think it's a big problem, but I just ran into an issue playing
around with vagrant. I was using the 0.9.0 github branch to run kafka, and
used vagrant to (by default) bring up one zookeeper and 3 broker instances.
Then I created two topics like:
./bin/kafka-topics.sh --create --zookeeper 1
Hi all,
I'm running all little test, with both zookeeper, Kafka and the schema
registry running locally. Using the new consumer, and the 2.0.0-snapshot
version of the registry, which has an decoder giving back instances of the
schema object.
It's all working fine, but I see a consistent delay maxi
I just ran in almost the same problem. In my case it was solved by setting
the 'advertised.host.name' to the correct value in de server properties.
The hostname you enter here should be resolvable from the cluster your
running the test from.
On Mon, Nov 30, 2015 at 3:40 AM Yuheng Du wrote:
> Als
I was experimenting with the timeout setting, but as long as messages are
produced and the consumer(s) keep polling I saw little difference. I did
see for example that when producing only 1 message a second, still it
sometimes wait to get three messages. So I also would like to know if there
is a f
age are much larger than the default setting which is
> 1024 bytes but still the consumer doesn't return messages for every poll.
>
>
> On Tue, 1 Dec 2015 at 18:29 Gerard Klijs wrote:
>
> > I was experimenting with the timeout setting, but as long as messages are
> >
x27;m looking for any connection problems or metadata/fetch
> request
> > > errors. And I have to ask a dumb question, how do you know that more
> > > messages are available? Are you monitoring the consumer's lag?
> > >
> > > -Jason
> > >
>
Hi Suyog,
It working as intended. You could set the property min.fetch.bytes to a
small value to get less messages in each batch. Setting it to zero will
probably mean you get one object with each batch, at least was the case
when I tried, but I was producing and consuming at the same time.
On Tue
Interesting,
Do you have any way to look into the amount and size of the actual traffic
being send. I assume because there are more partitions the actual sending
of the messages is separated into more messages, leading to more overhead.
On Sat, Feb 20, 2016 at 12:57 PM John Yost wrote:
> Hi Eve
Hi Ted,
Maybe it's usefull to take a look at samza, http://samza.apache.org/ they
use kafka in a way which sounds similar to how you want to use it. As I
recall from a youtube conference the creator of samza also mentioned to
never delete the events. These things are off course very dependent on y
Don't know the actual question, it matters what you want to do.
Just watch out trying to copy every topic using a new consumer, cause then
internal topics are copied, leading to errors.
Here is a temple start script we used:
#!/usr/bin/env bash
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote
What do you see in the logs?
It could be it goes wrong because you have the bootstrap.servers property
which is not supported for the old consumer.
On Wed, Mar 9, 2016 at 11:05 AM Gerard Klijs
wrote:
> Don't know the actual question, it matters what you want to do.
> Just watch ou
Nice read. We just started using kafka, and have multiple cases which need
some kind of stream processing. So we most likely will start testing/using
it as soon as it will be released, adding stream processing containers to
our docker landscape.
On Fri, Mar 11, 2016 at 2:42 AM Jay Kreps wrote:
>
I noticed a similar effect with a test tool, which checked if the order the
records were produced in, was the same as the order in which they were
consumed. Using only one partition it works fine, but using multiple
partitions the order gets messed up. If I'm right this is by design, but I
would li
You might find what you want when looking how Kafka is used for samza,
http://samza.apache.org/
On Mon, Mar 14, 2016 at 10:34 AM Daniel Schierbeck
wrote:
> Partitions being limited by disk size is no different from e.g. a SQL
> store. This would not be used for extremely high throughput. If,
> e
Hi, if you use a new group for a consumer, the auto.offset.reset value will
determine whether it will start at the beginning (with value earliest) or
at the end (with value latest). For each group a separate offset is used,
to two consumer, belonging to two different groups, when started before the
In addition, because we also use the acl, creating a lot of topics is
cumbersome. So in one of our tests I added an uuid to the message so I know
which message was produced for a certain test.
On Mon, Mar 14, 2016 at 11:15 PM Stevo Slavić wrote:
> See
>
> https://cwiki.apache.org/confluence/disp
I think Kafka at the moment is not mature enough to support a LTS release.
I think it will take a lot of effort to 'guarantee' a back-port will be
more safe to use in production then the new release. For example, when you
will manage the release of 0.9.0.2, with the fixes from 0.10.0.0, you need
to
t; with?
>
> On Mon, Mar 21, 2016 at 4:49 PM, Gerard Klijs
> wrote:
>
> > I think Kafka at the moment is not mature enough to support a LTS
> release.
> > I think it will take a lot of effort to 'guarantee' a back-port will be
> > more safe to use in pro
Hi Mark,
I don't think it would be a good solution with the latencies to and from
the server your running from in mind. This is less of a problem is your app
is only mainly used in one region.
I recently went to a Firebase event, and it seems a lot more fitting. It
also allows the user to see it'
roduction for critical use-cases. I think it
> > is
> > > already *expected *to work as per the theory and any bugs need to be
> > > patched. As there is no one patching the older releases and the
> companies
> > > refusing to upgrade due to the way enter
I only have experience with option 1. In this case it's simple. You provide
the location of the keystore in the properties, so you can use multiple
certificates for multiple clients. If you like this could even be in the
same application.
On Tue, Mar 22, 2016 at 3:13 AM Raghavan, Gopal
wrote:
>
If I'm reading right, your question is more about how to successfully
de(serialise) java object? You might want to take a look at the confluent
avro schema registry. Using avro schema's you can easily store messages in
a java object created by the schema. This way the messages will also be a
lot sm
The super user is indeed for the broker to be able to do all the things it
needs to do. For consumers and producers you can set the correct rights
with the acl tool. http://kafka.apache.org/documentation.html#security_authz
On Tue, Mar 22, 2016 at 8:28 PM christopher palm wrote:
> Hi Ismael,
>
>
If you don't specify the partition, and do have a key, then the default
behaviour is to use a hash on the key to determine the partition. This to
make sure the messages with the same key and up on the same partition. This
helps to ensure ordering relative to the key/partition. Also when using
compa
You could check what it does, and do that instead of relying in the script.
It runs the kafka.admin.AclCommand class with some properties, and sets
some jvm settings.
On Thu, Mar 31, 2016 at 4:36 PM Kalpesh Jadhav <
kalpesh.jad...@citiustech.com> wrote:
> Hi,
>
> Is there any java api available t
Don't know if adding it to Kafka is a good thing. I assume you need some
java opts settings for it to work, and with other solutions these would be
different. It could be enabled with an option off course, then it's not in
the way if you use something else.
We use zabbix, this is a single tool whic
Yes, but only via the broker you connect to.
On Mon, Apr 4, 2016, 07:10 Ratha v wrote:
> I'm not seeing such parameter as an input for consumer.
>
> Does version 0.9.x use zookeeper?
>
> --
> -Ratha
> http://vvratha.blogspot.com/
>
Is it an option to set up a cluster and kill the leader? That's the way we
checked retries and at if we would not lose messages that way.
The sending to Kafka goes in two parts, some serialization etc, before an
attempt is made to really send the binary message, and the actual sending.
I'm not sure
The options can only be used to set "The properties to initialize the
message formatter." You have several options, you could use different
properties fils, with only the group.id being different. Another option is
use a .properties.template, with a group.id=, and with a batch
script first set the
As far as I know the cientId is only used for logging, so you could set it
to whatever is most usefull in the logging. You might for example want to
use the ip as the id, so when you get errors you know where to look.
On Tue, Apr 19, 2016 at 6:51 PM Rick Rineholt wrote:
> Hi,
> If I have multipl
We used a dockerized zabbix, one of the advantages of zabbix is that it
has, jmx readout, creation of items, graphs, alerts in one product. Also
how long to keep the history can be set for each item. The interface is not
very intuitive though.
On Mon, Apr 25, 2016 at 10:14 AM Mudit Kumar wrote:
Using kafka streams is one way, I used camel before with kafka, which also
has a nice way of using filters.
On Fri, Apr 29, 2016 at 1:51 PM Subramanian Karunanithi
wrote:
> Hi,
>
> When a stream of data passes through Kafka, wanted to apply the filter and
> then let that message pass through to
I think by design it would be better to put different kind of messages in a
different topic. But if you would want to mix you can make your own
serializer/deserializer you could append a 'magic byte' to the byes you get
after you serialize, to be able to deserialize using the correct methods.
The c
Looks like it, you need to be sure the offset topic is using compaction,
and the broker is set to enable compaction.
On Tue, May 3, 2016 at 9:56 AM Jun MA wrote:
> Hi,
>
> I’m using 0.9.0.1 new-consumer api. I noticed that after kafka cleans up
> all old log segments(reach delete.retention time)
serializer for all type of the objects we publish? OR seperate
> for each objects?
> If we have seperate serializer/deserializers, then how can I configure
> kafka?
> Or Is it recommended to use Avro schemas?
>
> Thanks
>
> On 2 May 2016 at 18:43, Gerard Klijs wrote:
>
>
the better way to achieve this?
>
> On 3 May 2016 at 18:20, Gerard Klijs wrote:
>
> > If you put them in one topic, you will need one
> > 'master' serializer/deserializers which can handle all the formats.
> > I don't know how you would like to use Avro schem
Both are possible, but the 'new' consumer stores the offset in an __offset
topic.
On Tue, May 10, 2016 at 7:07 AM Snehalata Nagaje <
snehalata.nag...@harbingergroup.com> wrote:
>
>
> Hi All,
>
>
> As per kafka 9 version, where does kafka store committed offset?
>
> is it in zookeeper or kafka bro
You could create a docker image with a kafka installation, and start a
mirror maker in it, you could set the retention time for it to infinite,
and mount the data volume. With the data you could always restart the
docker, en mirror it to somewhere else. Not sure that's what you want, but
it's an op
I don't think it's possible since the offsets of both clusters can be
different, you don't know if it will work correctly. When I used the mirror
maker accidentally on the __consumer_offsets topic it also gave some
errors, so I don't know if it's technically possible. A possible future
solution wo
Depends on your use case but I guess something like this:
- Install al fresh on the new VM's
- Start a mirror maker in the the new VM's to copy data from the old ones
- Be sure it's working right
- Shut down the old VM's and start using the new ones
The last step is the trickiest and depends a lot
I was just trying to do the same thing, but it does not seem to support ssl
(or at least not in combination with the acl). I get similar errors as in
https://issues.apache.org/jira/browse/KAFKA-3151 but when I used
--command-config /home/kafka/consumer.properties to give the ssl properties
it worke
Are you sure consumers are always up, when they are behind they could
generate a lot of traffic in a small amount of time?
On Mon, May 23, 2016 at 9:11 AM Anishek Agarwal wrote:
> additionally all the read / writes are happening via storm topologies.
>
> On Mon, May 23, 2016 at 12:17 PM, Anishek
For both old and new consumers/producers you can make your own
(de)serializer to do some encryption, maybe that could be an option?
On Tue, May 24, 2016 at 2:40 PM Tom Crayford wrote:
> Hi,
>
> There's no encryption at rest. It's recommended to use filesystem
> encryption, or encryption of each
Compaction works on same keys, not same messages. So at a configured time
it will go through the log and delete al records but the most recent one
with the same key. I guess, but I'm not entirely sure the null message
get's deleted in the same pass, since it's a null message. So your example
would
It might be there never was/currently isn't a consumer with the group
jopgroup consuming from the twitter topic. I only used it for the new
consumer(offsets in broker), and then the group needs to be 'active' in
order to get the offsets.
On Mon, May 30, 2016 at 2:37 PM Diego Woitasen wrote:
> Hi
If you want system administrators not being able to see the data, the only
option is encryption, with only the clients sharing the key (or whatever is
used to (de)crypt the data). Like the example from eugene. I don't know the
kind of messages you have, but you could always wrap something around an
What do you have configured, do you have the brokers set as super users,
with the right certificate?
On Wed, Jun 1, 2016 at 6:43 AM 换个头像 wrote:
> Hi Kafka Experts,
>
>
> I setup a secured kafka cluster(slal-plain authentication). But when I try
> to add ACLs for some existing topics, all three b
If I understand it correctly each consumer should have it's 'own' thread,
and should not be accessible from other threads. But you could
(dynamically) create enough threads to cover all the partitions, so each
consumer only reads from one partition. You could also let all those
consumers access som
We use almost the same properties (the same if you account for defaults),
and have not seen any check whether the FQDN matches the CN, as it's al
working without matching names. It seems the requirement is only needed if
you use SASL_SSL as security protocol, which from you config you don't seem
to
munication. Looks
> like you are expecting to use the username "admin" for the broker, so you
> should set security.inter.broker.protocol=SASL_PLAINTEXT. There is also a
> missing entry in the KafkaServer section of jaas.conf. You need to add
> user_admin="welcome1
Although I think it should have been an included option, it's very easy to
create and use your own message handler with the mirror maker. You can
simply copy the timestamp and type from the consumerecord to the
producerecord.
On Wed, Jun 1, 2016 at 5:48 PM Gwen Shapira wrote:
> The intent was de
ly.
>
> > On 31 May 2016, at 13:00, Gerard Klijs wrote:
> >
> > If you want system administrators not being able to see the data, the
> only
> > option is encryption, with only the clients sharing the key (or whatever
> is
> > used to (de)crypt the data). Li
No you can't because:
- because of producer failures some messages may be duplicated.
- your not sure the cluster your copying from hasn't some already removed
data.
We try to solve the same problem, and are probably going to solve it by
copying the timestamps with the mirror maker, and on the swi
Not that I have anything against paying for monitoring, or against
Confluent, but you will need your consumers to be using kafka 1.10 is you
want to make most out of the confluent solution. We currently are using
zabbix, it's free, and it has complete functionality in one product. It
does can be a
First thing which comes to my mind, did you use it on a new kafka (broker)
version? Streams don't work with older brokers.
On Fri, Jun 3, 2016 at 9:15 AM 听风 <1429327...@qq.com> wrote:
> Hello，
>
>
> Can someone give me a hand?
>
>
> version: kafka_2.11-0.10.0.0
>
>
> Run kafka strams application
I asume you use a replication factor of 3 for the topics? When I ran some
test with producer/consumers in a dockerized setup, there where only few
failures before the producer switched to to correct new broker again. I
don't know the exact time, but seemed like a few seconds at max, this was
with w
I don't know about 'normal', we use the mockproducer with autocomplete set
to false, and use a responseThread to simulate produce behaviour like this:
private final class ResponseThread extends Thread {
public void run() {
try {
Thread.sleep(responseTime);
} catch
If you can put the acl in a file, and there will be little or none changes,
you might be best of writing your own Authorizer implementation. If you can
used a shared file system to store the config you would even be able to
easily change it, and it will be the same across the cluster.
On Thu, Jun
It's correct that's that's that what needs te be done. But usually you let
the serializer do that for your, if there is no way te set a serializer,
you probably could do it this way, and the message can be read by a regular
kafka avro consumer.
On Mon, Jun 13, 2016 at 5:45 PM Tauzell, Dave
wrote:
You could try set the acks to -1, so you wait for the produce to be
succesfull, until most other brokers also received the message. Another
thing you could try is set the unclean.leader.election.enable to false
(this is a setting on the broker).
I think what's happening now is that the message in y
+1 we already use java 8
On Fri, Jun 17, 2016 at 11:07 AM Jaikiran Pai
wrote:
> +1 for Java 8. Our eco-system which uses Kafka and many other open
> source projects are now fully on Java 8 since a year or more.
>
> -Jaikiran
> On Friday 17 June 2016 02:15 AM, Ismael Juma wrote:
> > Hi all,
> >
>
What do you mean with a *docker volume*? It's best to use a data container,
and use the volumes in your broker container, this way you can destroy the
broker container without affecting the data. The data container itself
needs to be configured depending on the host. For example when the host is
ru
No, why would you want to store the offsets in zookeeper? One of the
improvements is to not depend on zookeeper for the offsets. And there is
tooling to get the offsets (although the consumer group must exist).
On Mon, Jun 20, 2016 at 10:57 PM Bryan Baugher wrote:
> Hi everyone,
>
> With the new
You need to pass the correct options, similar to how you would do to a
client. We use the consumer-groups in a docker container, in an environment
witch is now only SSL (since the schema registry now supports it).
On Wed, Jun 22, 2016 at 2:47 PM Radu Radutiu wrote:
> Hi,
>
> Is is possible to co
container, having
access to the same certificate as the broker (so we don't need to set acl's
to use the tool).
On Wed, Jun 22, 2016 at 2:47 PM Gerard Klijs
wrote:
> You need to pass the correct options, similar to how you would do to a
> client. We use the consumer-groups in a docker
If your producer has acks set to 0, or if the retries is set to 0, in the
properties, it will be lost, else it will most likely be retried and send
to the new leader.
On Thu, Jun 23, 2016 at 2:53 AM Saeed Ansari wrote:
> Hi,
> I searched a lot for my question and I did not find a good answer may
Please follow the instructions here
> > http://kafka.apache.org/documentation.html#security_ssl . At
> > the end of the SSL section we've an example for produce and
> > consumer command line tools to pass in ssl configs.
> >
> > Thanks,
&
With 3 nodes, I assume you mean 3 clusters? If I understand correctly, say
you have 3 clusters, A, B, and C, you simultaneously:
- want to copy from A and B to C, to get an aggregation in C
- want to copy fram A and C to B, to get a fail-back aggregation in B.
Now what will happen when a message is
[2016-06-28 11:56:46,649] WARN Error while fetching metadata with
> correlation id 2 : {topictest4=LEADER_NOT_AVAILABLE}
> (org.apache.kafka.clients.NetworkClient)
>
> Is there any reason not all of the messages made it through? Is there a way
> to reset the offset so it trie
You could set the auto.commit.interval.ms to a lower value, in your example
it is 10 seconds, which can be a lot of messages. I don't really see how it
could be prevented any further, since offset's can only committed by
consumer to the partitions they are assigned to. I do believe there is some
wo
Another option is to set enable.auto.commit=false and never commit the
offset, it should always go back to latest that way.
On Tue, Jul 12, 2016 at 1:34 PM Michael Noll wrote:
> To explain what you're seeing: After you have run a consumer application
> once, it will have stored its latest consu
Are you sure the topic itself has indeed 1 partition?
If so the only partition should be matched to either one till some
error/rebalance occurs, does this indeed happen (a lot)?
On Wed, Jul 13, 2016 at 7:19 AM BYEONG-GI KIM wrote:
> Hello.
>
> I'm not sure whether it's a bug or not, but here is
You can't you only get a guarantee on the order for each partition, not
over partitions. Adding partitions will possible make it a lot worse, since
items with the same key wll land in other partitions. For example with two
partitions these will be about the hashes in each partitions:
partition-0: 0
Things like consumer rebalances on the cluster you copy from, and brokers
going down on the cluster your writing down can cause duplications. The
default settings are set to prevent data loss, making data duplication more
likely to happen in case of error.
You could possibly make a simple consumer
No, if you don't use streams you don't need them. If you have no clients
(so also no mirror maker) running on the same machine you also don't need
the client jar, if you run zookeeper separately you also don't need those.
On Fri, Jul 29, 2016 at 4:22 PM Bhuvaneswaran Gopalasami <
bhuvanragha...@gm
As I under stood it won't really has any advantage over using HDD since
most things will work from the working memory anyway. You might want to use
SSD for zookeeper through.
On Fri, Jul 29, 2016 at 12:19 AM Kessiler Rodrigues
wrote:
> Hi guys,
>
> Should I use SSD for my brokers or not?
>
> Wha
The default partitioner will take the key, make the hash from it, and do a
modulo operation to determine the partition it goes to. Some things which
might cause it to and up different for different topics:
- partition number are not the same (you already checked)
- key is not exactly the same, for
I don't know the answer to the second question, if you don't use (much)
auto-generated id's for the consumer group you should be ok, since it's a
compacted topic after all, you might want to check if the compaction is on.
We set the offsets.retention.minutes to a week without a problem.
On Tue, Au
With a linger of 5 seconds, 2-3 seconds would make sense when the load is
smaller, are are sure the measurements with 0.8.2.1 where with the same
load and/or linger worked correctly there?
On Fri, Sep 2, 2016 at 1:12 AM Yifan Ying wrote:
> We tried to upgrade the Kafka clients dependency from 0.
This is a known bug, I think it was fixed in the 0.10.0.1 release. You
could alternatively use a custom message handler for the mirror maker, and
then use the produce without a timestamp when the timestamp is -1 in the
consuming message.
On Thu, Sep 15, 2016 at 9:48 AM Samuel Zhou wrote:
> Hi,
>
We just had an interesting issue, luckily this was only on our test cluster.
Because of some reason one of the machines in a cluster became really slow.
Because it was still alive, it stil was the leader for some
topic-partitions. Our mirror maker reads and writes to multiple
topic-partitions on ea
.
>
>
> As far as mirror maker goes, you can play with the consumer/producer
> timeout settings to make sure the process waits long enough for a slow
> machine.
>
> -David
>
> On 9/16/16, 7:11 AM, "Gerard Klijs" wrote:
>
> We just had an interesting i
We have a simple application producing 1 msg/sec, and did nothing to
optimise the performance and have about a 10 msec delay between consumer
and producer. When low latency is important, maybe pulsar is a better fit,
https://www.datanami.com/2016/09/07/yahoos-new-pulsar-kafka-competitor/ .
On Tue,
benchmarks on this throughput
> wise?
>
>
>
>
>
>
> On Thu, Sep 22, 2016 7:45 AM, Gerard Klijs gerard.kl...@dizzit.com
> wrote:
> We have a simple application producing 1 msg/sec, and did nothing to
>
> optimise the performance and have about a 10 msec delay between co
message to
> write ahead log so messages are never lost.
> My question now is that what settings I need to change in Kafka so it will
> store every message? that way I am comparing apples to apples.
>
>
>
>
>
>
> On Fri, Sep 23, 2016 12:06 AM, Gerard Klijs gerard.kl...@d
I think, but don't know for sure, it doesn't matter for consumers, since
the messages you read are still 'old' images. I would expect errors van you
use an old producer, and/or when consuming the record from the old producer.
On Mon, Oct 3, 2016 at 7:09 AM Nikhil Goyal wrote:
> Hi guys,
>
> I cr
You could limit the access to zookeeper, with kerberos, or with a firewall.
For example to only allow connections to zookeeper from the cluster itself,
this way you need to access those machines to be able to set acls. The
create permission is used for creating topics I think, there is no acl to
li
There's already a pretty active Kafka Meetup group 'Kafka Utrecht' which
had meetup in Amsterdam and Rotterdam in the past.
Op di 12 mrt. 2019 om 06:37 schreef Antoine Laffez <
antoine.laf...@lunatech.nl>:
> Hi!
> The workshop is an internal workshop given by certified trainer employee.
> It is 2
Not really possible as the producer assumes you are using the schema
registry. You can use avro for the deserialisation in some other way, but
you need to create (de)serializers that fit with the other way.
Op wo 27 mrt. 2019 om 17:33 schreef lsroudi abdel :
> It depend on your use case, you coul
Most likely something went wrong creating the keystores, causing the SSL
handshake to fail. Its important to have a valid chain, from the
certificate in the struststore, and then maybe intermediates tot the
keystore.
On Fri, Dec 16, 2016, 00:32 Raghu B wrote:
Thanks Derar & Kiran, your suggestio
Not really, as you can update the schema, and have multiple of them at the
same time. By default each schema has to backwards compatible, so you do
have to exclude the specific topic you use with different schema's. With
every write, the 'id' of the schema used is also written, so when you
deserial
92 matches
Mail list logo
