Re: Kafka Streams new release

2018-07-11 Thread Ofir Manor
>From the release plan:
  " While the target release date is fixed at ~2w after code freeze, RCs
will roll out as needed until the release vote passes"
   https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80448820
You can follow the voting threads on this mailing list - the current thread
is "[VOTE] 2.0.0 RC2"
Once a vote for RC passes, that RC will be released as the 2.0.0 version.

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

On Wed, Jul 11, 2018 at 9:21 AM, Ayushi Sharma  wrote:

> When is the next Kafka Streams release (Kafka Streams 2.0.0) which was
> tentatively on 26June?
>


Re: Allow to replace Zookeeper with different Coordination Service (etcd/consul)

2017-03-06 Thread Ofir Manor
I suggest you check KIP-30 and the extensive discussion about it in the
mailing list  from around December 2015 called "[DISCUSS] KIP-30 Allow for
brokers to have plug-able consensus and meta data storage sub systems"
If I remember correctly, it ran into some objections, as the existing
commiters thought at the time that ZK was significantly more reliable, so
it was not worth the effort to add inferior options.
I personally think nowadays, when a lot of other critical cluster infra
relies on these coordination services anyway, this KIP makes a lot of
sense. The current dependency of ZK creates a large, unneeded operational
overhead for those who have already deployed and relies on etcd/consul for
the rest of their stack (including other stateful services).
Just my two cents,

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

On Mon, Mar 6, 2017 at 12:15 PM, Alexander Binzberger <
alexander.binzber...@wingcon.com> wrote:

> I would also be interested in this. (etcd)
>
>
> Am 02.03.2017 um 12:24 schrieb Molnár Bálint:
>
>> Hi,
>>
>> I was wonderring to refactor Kafka core code to be able to use different
>> Coordination Service than Zookeeper. I know I will need to create a KIP
>> for
>> that.
>> I think the first part of this task to refactor the classes which are
>> using
>> the ZkUtil methods to use a zookeeper independent trait instead.
>> After that I think it will be possible to replace Zookeeper with
>> etcd/Consul or even with a Raft implementation.
>> Even without additional implementation it would help to test the code
>> without starting an embedded zookeeper.
>> I have already started to implement a POC and it seems doable, even if
>> it's
>> not a small patch.
>>
>> Balint
>>
>>
> --
> Alexander Binzberger
> System Designer - WINGcon AG
> Tel. +49 7543 966-119
>
> Sitz der Gesellschaft: Langenargen
> Registergericht: ULM, HRB 734260
> USt-Id.: DE232931635, WEEE-Id.: DE74015979
> Vorstand: thomasThomas Ehrle (Vorsitz), Fritz R. Paul (Stellvertreter),
> Tobias Treß
> Aufsichtsrat: Jürgen Maucher (Vorsitz), Andreas Paul (Stellvertreter),
> Martin Sauter
>
>


Re: [DISCUSS] 0.10.3.0/0.11.0.0 release planning

2017-02-28 Thread Ofir Manor
Thanks Ismael, makes perfect sense :)

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

On Tue, Feb 28, 2017 at 1:02 PM, Ismael Juma <ism...@juma.me.uk> wrote:

> Hi Ofir,
>
> Thanks for the feedback. This will be the first release with exactly-once.
> I think we should give ourselves at least one stabilisation/polish/feedback
> cycle before we go for 1.0. :)
>
> Ismael
>
> On Tue, Feb 28, 2017 at 10:55 AM, Ofir Manor <ofir.ma...@equalum.io>
> wrote:
>
> > This is definitely a major release, looks already quite exciting...
> > I don't want to start a bike-shading argument, but a few people have told
> > me in the past year that once exactly-once delivery lands, Kafka would
> > likely bump to 1.0. I do like it, but don't feel strongly either way.
> >
> > Ofir Manor
> >
> > Co-Founder & CTO | Equalum
> >
> > Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
> >
> > On Tue, Feb 28, 2017 at 12:37 PM, Ismael Juma <ism...@juma.me.uk> wrote:
> >
> > > Hi Vahid,
> > >
> > > Sure, I've added KIP-54. I thought I had done it, sorry for the
> > oversight.
> > >
> > > Ismael
> > >
> > > On Tue, Feb 28, 2017 at 4:31 AM, Vahid S Hashemian <
> > > vahidhashem...@us.ibm.com> wrote:
> > >
> > > > +1 on 0.11.0.0.
> > > >
> > > > Can we also include KIP-54 to the list?
> > > > The PR for this KIP is ready for review.
> > > >
> > > > Thanks.
> > > > --Vahid
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > From:   Ismael Juma <ism...@juma.me.uk>
> > > > To: dev@kafka.apache.org
> > > > Date:   02/27/2017 07:47 PM
> > > > Subject:[DISCUSS] 0.10.3.0/0.11.0.0 release planning
> > > > Sent by:isma...@gmail.com
> > > >
> > > >
> > > >
> > > > Hi all,
> > > >
> > > > With 0.10.2.0 out of the way, I would like to volunteer to be the
> > release
> > > > manager for our next time-based release. See
> > https://cwiki.apache.org/c
> > > > onfluence/display/KAFKA/Time+Based+Release+Plan if you missed
> previous
> > > > communication on time-based releases or need a reminder.
> > > >
> > > > I put together a draft release plan with June 2017 as the release
> month
> > > > (as
> > > > previously agreed) and a list of KIPs that have already been voted:
> > > >
> > > > *https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=68716876
> > > > <https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=68716876
> > > > >*
> > > >
> > > > I haven't set exact dates for the various stages (feature freeze,
> code
> > > > freeze, etc.) for now as Ewen is going to send out an email with some
> > > > suggested tweaks based on his experience as release manager for
> > 0.10.2.0.
> > > > We can set the exact dates after that discussion.
> > > >
> > > > As we are starting the process early this time, we should expect the
> > > > number
> > > > of KIPs in the plan to grow (so don't worry if your KIP is not there
> > > yet),
> > > > but it's good to see that we already have 10 (including 2 merged and
> 2
> > > > with
> > > > PR reviews in progress).
> > > >
> > > > Out of the KIPs listed, KIP-98 (Exactly-once and Transactions) and
> > > KIP-101
> > > > (Leader Generation in Replication) require message format changes,
> > which
> > > > typically imply a major version bump (i.e. 0.11.0.0). If we do that,
> > then
> > > > it makes sense to also include KIP-106 (Unclean leader election
> should
> > be
> > > > false by default) and KIP-118 (Drop support for Java 7). We would
> also
> > > > take
> > > > the chance to remove deprecated code, in that case.
> > > >
> > > > Given the above, how do people feel about 0.11.0.0 as the next Kafka
> > > > version? Please share your thoughts.
> > > >
> > > > Thanks,
> > > > Ismael
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>


Re: [DISCUSS] 0.10.3.0/0.11.0.0 release planning

2017-02-28 Thread Ofir Manor
This is definitely a major release, looks already quite exciting...
I don't want to start a bike-shading argument, but a few people have told
me in the past year that once exactly-once delivery lands, Kafka would
likely bump to 1.0. I do like it, but don't feel strongly either way.

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

On Tue, Feb 28, 2017 at 12:37 PM, Ismael Juma <ism...@juma.me.uk> wrote:

> Hi Vahid,
>
> Sure, I've added KIP-54. I thought I had done it, sorry for the oversight.
>
> Ismael
>
> On Tue, Feb 28, 2017 at 4:31 AM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
> > +1 on 0.11.0.0.
> >
> > Can we also include KIP-54 to the list?
> > The PR for this KIP is ready for review.
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> >
> >
> > From:   Ismael Juma <ism...@juma.me.uk>
> > To: dev@kafka.apache.org
> > Date:   02/27/2017 07:47 PM
> > Subject:[DISCUSS] 0.10.3.0/0.11.0.0 release planning
> > Sent by:isma...@gmail.com
> >
> >
> >
> > Hi all,
> >
> > With 0.10.2.0 out of the way, I would like to volunteer to be the release
> > manager for our next time-based release. See https://cwiki.apache.org/c
> > onfluence/display/KAFKA/Time+Based+Release+Plan if you missed previous
> > communication on time-based releases or need a reminder.
> >
> > I put together a draft release plan with June 2017 as the release month
> > (as
> > previously agreed) and a list of KIPs that have already been voted:
> >
> > *https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=68716876
> > <https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=68716876
> > >*
> >
> > I haven't set exact dates for the various stages (feature freeze, code
> > freeze, etc.) for now as Ewen is going to send out an email with some
> > suggested tweaks based on his experience as release manager for 0.10.2.0.
> > We can set the exact dates after that discussion.
> >
> > As we are starting the process early this time, we should expect the
> > number
> > of KIPs in the plan to grow (so don't worry if your KIP is not there
> yet),
> > but it's good to see that we already have 10 (including 2 merged and 2
> > with
> > PR reviews in progress).
> >
> > Out of the KIPs listed, KIP-98 (Exactly-once and Transactions) and
> KIP-101
> > (Leader Generation in Replication) require message format changes, which
> > typically imply a major version bump (i.e. 0.11.0.0). If we do that, then
> > it makes sense to also include KIP-106 (Unclean leader election should be
> > false by default) and KIP-118 (Drop support for Java 7). We would also
> > take
> > the chance to remove deprecated code, in that case.
> >
> > Given the above, how do people feel about 0.11.0.0 as the next Kafka
> > version? Please share your thoughts.
> >
> > Thanks,
> > Ismael
> >
> >
> >
> >
> >
>


Re: Restrict consumers from connecting to Kafka cluster

2016-11-22 Thread Ofir Manor
Check the Security section of the documentation, especially authorization
(which means also authentication)
http://kafka.apache.org/documentation.html

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

On Tue, Nov 22, 2016 at 9:13 PM, ravi singh <rrs120...@gmail.com> wrote:

> Is it possible to restrict Kafka consumers from consuming from a given
>  Kafka cluster?
>
> --
> *Regards,*
> *Ravi*
>


Re: [VOTE] Add REST Server to Apache Kafka

2016-10-26 Thread Ofir Manor
-1
First, I think there was no technical discussion of the KIP details in the
DISCUSS until now, just a binary yes/no. Real discussion just started with
the excellent ten technical points from Ewen a few hours ago.
Generally,I don't see how a standard REST makes sense for format-agnostic
realtime streaming. It would likely be a poor choice for any high-volume
messaging, and encouraging its usage instead of the existing native clients
(or promoting it as an equivalent alternative) is likely a mistake and will
hurt users.
For those needing that functionality, they can get it today and can also
participate in that ecosystem project (or fork it).

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

On Wed, Oct 26, 2016 at 12:16 AM, Harsha Chintalapani <ka...@harsha.io>
wrote:

> Hi All,
>We are proposing to have a REST Server as part of  Apache Kafka
> to provide producer/consumer/admin APIs. We Strongly believe having
> REST server functionality with Apache Kafka will help a lot of users.
> Here is the KIP that Mani Kumar wrote
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 80:+Kafka+Rest+Server.
> There is a discussion thread in dev list that had differing opinions on
> whether to include REST server in Apache Kafka or not. You can read more
> about that in this thread
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201610.mbox/%3CCAMVt_
> aymqeudm39znsxgktpdde46sowmqhsxop-+jmbcuv7...@mail.gmail.com%3E
>
>   This is a VOTE thread to check interest in the community for
> adding REST Server implementation in Apache Kafka.
>
> Thanks,
> Harsha
>


Re: [DISCUSS] KIP-80: Kafka REST Server

2016-10-06 Thread Ofir Manor
I personally think it would be quite wasteful to re-implement the REST
gateway just because that an actively-maintained piece of Apache-licensed
software is not governed directly by the Apache Kafka community... While
kafka-rest repo is owned by Confluent, the contributors including the main
one are also part of the Apache Kafka  community, so there is a chance to
work this out.

However, there are two valid concerns here that could be addressed, around
community and accessibility:
>> What we are worried about is a project
>> that's not maintained in a community. So the process of accepting patches
>> and priorities is not clear, and it's not developed in Apache community.
>> Not only that, existing REST API project doesn't support new client API
and
>> hence there is no security support either.

This might be easy to fix. Maybe Confluent / kafka-rest community can
clarify that - what is their contribution policy, dev style, roadmap etc.
If they want, they can make an effort to encourage participation from
people outside Confluent (easily accept contributions, invite external
commiters or have open dev process similar to Apache Kafka etc), as there
is definitely seems to be some interest on the list. That might clear the
community concern and help kafka-rest project (but that is a calculation
Confluent will have to make).

The other, independent, concern is that REST is something that is expected
to be available out of the box with Kafka. I personally don't feel strongly
about it (better use proper, efficient APIs from day one), though it is
definitely way smaller than adding a stream processing engine to the
project :)
Again,the kafka-rest "community" could take steps to make it even easier to
install, configure and run kafka-rest for new users on vanilla Apache Kafka
(outside the Confluent platform), if they wish that (or welcome
contributions to that end), but that is up to them.
Finally, if after the above steps were taken there would still a strong
desire to include a great rest gateway with Apache Kafka, I assume the
community could hypothetically fork the existing kafka-rest into an Apache
Kafka subproject and maintain it "within Apache" instead of implementing it
from scratch (though I'm not a lawyer etc) - but I cannot imagine it happen
without Confluent blessing, and I think that is likely much less optimal
(pulling in other Confluent / Apache licensed dependencies) than having a
separate external community around kafka-rest.


Just my two cents,


Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <ka...@harsha.io>
wrote:

> Neha & Jay,
>  We did look at the open source alternatives. Our concern
> is what's the patch acceptance and adding features/ bug-fixes to the
> existing project under a Github (although it's licensed under Apache 2.0).
> It would be great if that project made available under Apache and driven by
> the community.  Adding to the above, not all Kafka users are interested in
> using the Java client API, they would like to have simple REST API where
> they can code against using any language. I do believe this adds value to
> Apache Kafka in itself.
>
> "For 1, I don't think there is value in giving in to the NIH syndrome and
> reinventing the wheel. What I'm looking for is a detailed comparison of the
> gaps and why those can't be improved in the REST proxy that already exists
> and is actively maintained."
>
> We are not looking at this as  NIH. What we are worried about is a project
> that's not maintained in a community. So the process of accepting patches
> and priorities is not clear, and it's not developed in Apache community.
> Not only that, existing REST API project doesn't support new client API and
> hence there is no security support either.
> We don't know the timeline when that's made available. We would like to add
> admin functionality into the REST API. So the Roadmap of that project is
> not driven by Apache.
>
>
> "This doesn't materially have an impact on expanding the usability of
>Kafka. In my experience, REST proxy + Java clients only cover ~50% of
> all
>Kafka users, and maybe 10% of those are the ones who will use the REST
>proxy. The remaining 50% are non-java client users (C, python, go, node
>etc)."
>
> REST API is most often asked feature in my interactions with Kafka users.
> In an organization, There will be independent teams who will write their
>  Kafka clients using different language libraries available today, and
> there is no way to standardize this. Instead of supporting several
> different client libraries users will be interested in using a REST API
> server. The need for a REST API will only increase as more and 

Re: [DISCUSS] Time-based releases for Apache Kafka

2016-08-25 Thread Ofir Manor
I happily agree that Kafka is a solid and the community is great :)
But I think there is a gap in perception here.
For me, LTS means that someone is actively taking care of a release -
actively backporting critical fixes (security, stability, data loss,
corruption, hangs etc) from trunk to that LTS version periodically for an
extended period of time, for example 18-36 months... So people can really
rely on the same Kafka version for a long time.
Is someone doing it today for 0.9.0? When is 0.9.0.2 expected? When is
0.8.2.3 expected? Will they cover all known critical issues for whoever
relies on them in production?
In other words, what is the scope of support that the community want to
commit for older versions? (upgrade compatibility? investigating bug
reports? proactively backporting fixes?)
BTW, another legit option is that the Apache Kafka project won't commit to
LTS releases. It could let commercial vendors compete on supporting very
old versions. I find that actually quite reasonable as well.

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

On Thu, Aug 25, 2016 at 8:19 PM, Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:

> I agree that the Kafka community has managed to maintain a very high
> quality level, so I'm not concerned
> about the quality of non-LTS releases. If the principle is that every
> release is supported for 2 years, that
> would be good. I suppose that if the burden of having that many in-support
> releases proves too heavy,
> as you say we could reconsider.
>
> Andrew Schofield
>
> 
> > From: g...@confluent.io
> > Date: Thu, 25 Aug 2016 09:57:30 -0700
> > Subject: Re: [DISCUSS] Time-based releases for Apache Kafka
> > To: dev@kafka.apache.org
> >
> > I prefer Ismael's suggestion for supporting 2-years (6 releases)
> > rather than have designated LTS releases.
> >
> > The LTS model seems to work well when some releases are high quality
> > (LTS) and the rest are a bit more questionable. It is great for
> > companies like Redhat, where they have to invest less to support few
> > releases and let the community deal with everything else.
> >
> > Until now the Kafka community has managed to maintain very high
> > quality level. Not just for releases, our trunk is often of better
> > quality than other project's releases - we don't think of stability as
> > something you tuck into a release (and just some releases) but rather
> > as an on-going concern. There are costs to doing things that way, but
> > in general, I think it has served us well - allowing even conservative
> > companies to run on the latest released version.
> >
> > I hope we can agree to at least try maintaining last 6 releases as LTS
> > (i.e. every single release is supported for 2 years) rather than
> > designate some releases as better than others. Of course, if this
> > totally fails, we can reconsider.
> >
> > Gwen
> >
> > On Thu, Aug 25, 2016 at 9:51 AM, Andrew Schofield
> > <andrew_schofield_j...@outlook.com> wrote:
> >> The proposal sounds pretty good, but the main thing currently missing
> is a proper long-term support release.
> >>
> >> Having 3 releases a year sounds OK, but if they're all equivalent and
> bugfix releases are produced for the most
> >> recent 2 or 3 releases, anyone wanting to run on an "in support"
> release of Kafka has to upgrade every 8-12 months.
> >> If you don't actually want anything specific from the newer releases,
> it's just unnecessary churn.
> >>
> >> Wouldn't it be better to designate one release every 12-18 months as a
> long-term support release with bugfix releases
> >> produced for those for a longer period of say 24 months. That halves
> the upgrade work for people just wanting to keep
> >> "in support". Now that adoption is increasing, there are plenty of
> users that just want a dependable messaging system
> >> without having to be deeply knowledgeable about its innards.
> >>
> >> LTS works nicely for plenty of open-source projects. I think it would
> work well for Kafka too.
> >>
> >> Andrew Schofield
> >>
> >> 
> >>> From: ofir.ma...@equalum.io
> >>> Date: Thu, 25 Aug 2016 16:07:07 +0300
> >>> Subject: Re: [DISCUSS] Time-based releases for Apache Kafka
> >>> To: dev@kafka.apache.org
> >>>
> >>> Regarding bug fixes, you may want to consider to have an LTS release
> once a
> >>> year - designating it for longer-term support / better for 

Re: [DISCUSS] Time-based releases for Apache Kafka

2016-08-25 Thread Ofir Manor
Regarding bug fixes, you may want to consider to have an LTS release once a
year - designating it for longer-term support / better for the masses.
If you like that - then fix bugs in trunk, backport important ones to
latest release + the last two LTS releases.
Even if you don't - if a downstream distribution picks a Kafka version and
plans to support it over a few years, it could be nice of them to "own"
that older release - volunteer to be a release manager for bug backports to
that version over a longer period...
Just my two cents :)

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

On Thu, Aug 25, 2016 at 12:32 PM, Ismael Juma <ism...@juma.me.uk> wrote:

> Thanks for putting this together Gwen. I think it sounds reasonable and
> instead of trying to optimise every aspect of it ahead of time (which is
> hard, subjective and time-consuming), I am happy to try what is being
> proposed and tweak based on experience. One thing we should pay particular
> attention to is how the stabilisation process works out in practice.
>
> A couple of comments:
>
> "Given 3 releases a year and the fact that no one upgrades three times a
> year, we propose making sure (by testing!) that rolling upgrade can be done
> from each release in the past year (i.e. last 3 releases) to the latest
> version."
>
> Because the cost of doing this for a larger number of releases is
> relatively low, I still think we should go for 6 here (our code currently
> supports 5 versions as I said in a previous message, so we're close to that
> target already). I'm generally very keen to make upgrades as easy as
> possible so that people have no reason not to upgrade. :)
>
> "We will also attempt, as a community to do bugfix releases as needed for
> the last 3 releases."
>
> I would suggest 2, personally, but since this is a bit fuzzy, I am OK with
> 3 if people prefer that.
>
> Ismael
>
> On Thu, Aug 25, 2016 at 6:22 AM, Gwen Shapira <g...@confluent.io> wrote:
>
> > Hi Team Kafka,
> >
> > As per the KIP meeting, I created a wiki:
> > https://cwiki.apache.org/confluence/display/KAFKA/Time+
> Based+Release+Plan
> > Summarizing most of the discussion so far.
> >
> > Comments and additional discussion is welcome :)
> >
> > Gwen
> >
> > On Wed, Aug 17, 2016 at 12:31 PM, Vahid S Hashemian
> > <vahidhashem...@us.ibm.com> wrote:
> > > Time-based releases is a good idea and something that has proved to be
> > > working in a number of open source projects. One successful example is
> > > Node.js, that goes through two major releases a year. The interesting
> > fact
> > > about the two releases is that only one (the even-number release) comes
> > > with a long term support (LTS) plan (30 months). More can be read here:
> > > https://github.com/nodejs/LTS. The odd-number releases still come with
> > > major changes and help build the ecosystem, but as far as LTS goes,
> there
> > > is only one per year. This LTS plan makes most enterprises want to
> stick
> > > to even-number releases, which is okay since frequent upgrades is not
> > > something they are normally interested in anyway.
> > >
> > > There could be several minor releases (non-breaking) in between major
> > > releases. A major release contains all the features / bug fixes in the
> > > master branch a month before the release date, with the potential
> > addition
> > > of (non-breaking) bug fixes until the release day. The deprecation
> cycle
> > > is one major release: any functionality that is decided to be removed
> is
> > > deprecated in the next major release, and removed in the major release
> > > after that.
> > >
> > > Because of the success of LTS models in this and other open source
> > > projects, I would suggest implementing a formal LTS plan for Kafka too.
> > >
> > > Regards,
> > > --Vahid
> > >
> > >
> > >
> > > From:   Gwen Shapira <g...@confluent.io>
> > > To: dev@kafka.apache.org
> > > Date:   08/09/2016 04:49 PM
> > > Subject:[DISCUSS] Time-based releases for Apache Kafka
> > >
> > >
> > >
> > > Dear Kafka Developers and Users,
> > >
> > > In the past, our releases have been quite unpredictable. We'll notice
> > > that a large number of nice features made it in (or are close),
> > > someone would suggest a release and we'd do it. This is fun, but makes
> > > planning really hard - we saw it during the last release which we

Re: Feature request - Ability to use a network interface name instead of hostname/IP for Kafka server to use advertised hosts

2016-01-20 Thread Ofir Manor
Hi Jaikiran,
maybe this would help. I use Kafka inside VirtualBox, with no DNS and
potentially changing IP when guest is started.
So, in my Kafka startup script that runs on boot, I just replace the
advertised.host.name with the current external IP (it runs after network is
up):

sed -i "/^advertised.host.name/c\advertised.host.name=`hostname -I`"
$KAFKA_HOME/config/server.properties
nohup kafka-server-start.sh $KAFKA_HOME/config/server.properties
>/dev/null

Does that work for you?

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

On Wed, Jan 20, 2016 at 3:20 PM, Jaikiran Pai <jai.forums2...@gmail.com>
wrote:

> We are using Kafka 0.9.0 on our dev setups. We see that the Kafka
> configurations currently expose a way to configure the host.name and/or
> advertised.host.name which state:
>
> # Hostname the broker will bind to. If not set, the server will bind to
> all interfaces
> #host.name=
>
> # Hostname the broker will advertise to producers and consumers. If not
> set, it uses the
> # value for "host.name" if configured.  Otherwise, it will use the value
> returned from
> # java.net.InetAddress.getCanonicalHostName().
> #advertised.host.name=
>
> The problem for us is that for our system where we use Kafka, we don't use
> a DNS server and the IP allocation/assignment to the machine is via DHCP.
> As a result we don't have a set DNS name nor know the IP address before
> hand. However, we do know the network interface name to which we want the
> Kafka server to be using. Would it be possible add a feature where the
> config property could expose something like a advertised.host.interface
> which takes the name of the network interface which the Kafka server will
> use? Internally the implementation could then use any of the IP addresses
> returned by
> http://docs.oracle.com/javase/7/docs/api/java/net/NetworkInterface.html#getInterfaceAddresses%28%29
> after looking up the NetworkInterface by name
> http://docs.oracle.com/javase/7/docs/api/java/net/NetworkInterface.html#getByName%28java.lang.String%29
>
>
>
> Is this something that could be added as an enhancement?
>
> -Jaikiran
>
>