Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

[Randall Hauch]

Sonke,

Have you had a chance to update the KIP and kick off a VOTE thread? We need
to do this ASAP if we want this to make the KIP deadline for 1.1, which is
Jan 23!

On Tue, Jan 16, 2018 at 10:33 PM, Ewen Cheslack-Postava 
wrote:

> Sonke,
>
> I'm fine filtering some control characters. The trimming also seems like it
> might be *somewhat* moot because the way connector names work in standalone
> mode is limited by ConfigDef, which already does trimming of settings. Not
> a great reason to be restrictive, but we'd partly just be codifying what's
> there.
>
> I just generally have a distaste for being restrictive without a clear
> reason. In this case I don't think it has any significant impact.
>
> KIP freeze is nearing and this seems like a simple improvement and a PR is
> already available (modulo any changes re: control characters). I'll start
> reviewing the PR, do you want to make any last updates about control
> characters in the KIP and kick off a VOTE thread?
>
> -Ewen
>
> On Fri, Jan 12, 2018 at 1:43 PM, Colin McCabe  wrote:
>
> > On Fri, Jan 12, 2018, at 08:03, Sönke Liebau wrote:
> > > Hi everybody,
> > >
> > > from reading the discussion I understand that we have two things still
> > > open for discussen.
> > >
> > >  Ewen is still a bit on the fence about whether or not we trim
> > > whitespace characters but seems to favor not doing it due to there not
> > > being a real issue with them. I think it mostly boils down to a
> > > question of general preference. I am happy to change the code to allow
> > > leading and trailing whitespace characters again. If someone has a use
> > > case for these, so be it - I don't see a technical reason against
> > > them. Personally I think it is more likely that someone accidentally
> > > gets a whitespace character in his connector name somehow and
> > > subsequently has a confusing time figuring it out, but it shouldn't be
> > > that tough to spot and is correct behavior, so no issue with it.
> > > TL/DR: I'm happy either way :)
> > >
> > > Colin brought up control characters and that we should disallow these
> > > in connector names. The specific one that is mentioned in his link is
> > > Ascii 27 - ESC - \e so one possibility would be to explicitly
> > > blacklist this. The rest of the control characters (Ascii 0 through 31
> > > and 127) should be less critical I think, but since there is no way of
> > > knowing all software that might look at these strings and interpret
> > > them there is no real way of being certain. I think there is a case to
> > > be made for disallowing all control characters (and their respective
> > > escape sequences where applicable) in connector names - perhaps with
> > > the possible exclusion of /n /r /t .
> >
> > +1
> >
> > Colin
> >
> > >
> > > Thoughts?
> > >
> > > Kind regards,
> > > Sönke
> > >
> > >
> > >
> > > On Wed, Jan 10, 2018 at 7:23 AM, Ewen Cheslack-Postava
> > >  wrote:
> > > > great point, I'm always for exclusions where they make sense. i just
> > prefer
> > > > to include by default w/ exclusions when necessary to listing
> explicit
> > > > inclusions and being restrictive. (and security updates immediately
> as
> > > > needed).
> > > >
> > > > If you have a set of characters you think we should exclude, I think
> it
> > > > would be good to add them here or in a subsequent KIP!
> > > >
> > > > -Ewen
> > > >
> > > > On Tue, Jan 9, 2018 at 1:30 PM, Colin McCabe 
> > wrote:
> > > >
> > > >> On Sat, Jan 6, 2018, at 16:00, Ewen Cheslack-Postava wrote:
> > > >> > re: whitespace characters, I'm fine with the restriction since I
> > don't
> > > >> see
> > > >> > it becoming an issue in practice. I just don't see any reason to
> > restrict
> > > >> > it so it seems like we're going out of our way and doing extra
> work
> > to be
> > > >> > restrictive, but without clear motivation.
> > > >>
> > > >> There are very good reasons not to support control characters in
> file
> > > >> names, topic names, log files, etc.
> > > >>
> > > >> See http://seclists.org/fulldisclosure/2003/Feb/att-
> > 341/Termulation.txt
> > > >>
> > > >> There are a bunch of CVEs about this, too.  Because of the (in my
> > opinion,
> > > >> mistaken) decision to allow control characters in UNIX filenames,
> even
> > > >> echoing a file name to your terminal is a security vulnerability.
> > > >>
> > > >> best,
> > > >> Colin
> > > >>
> > > >>
> > > >> >
> > > >> > In general my default approach (without context of a specific
> > system)
> > > >> would
> > > >> > be to accept anything that we can encode in UTF-8 and only apply
> > > >> > restrictions where it becomes necessary (e.g. we need to define a
> > > >> delimiter
> > > >> > for some reason). The constraints of URLs introduce some
> complexity
> > (you
> > > >> > need escaping), but probably generally still allow this. If I can
> > use an
> > > >> > emoji when naming things, then I'm probably happy :) 

[VOTE] KIP-145: Expose Record Headers in Kafka Connect

[Randall Hauch]

Hi everyone,

I'd like to start the voting on this KIP to add support for headers in
Connect.:

*https://cwiki.apache.org/confluence/display/KAFKA/KIP-145+-+Expose+Record+Headers+in+Kafka+Connect
*

This does add a fair number of interfaces to our public API, and defines
some behavioral changes as well.

Thanks! Your feedback is highly appreciated.

Randall

Re: Kafka compacted topic question.

[Matthias J. Sax]

Yes and no.

There is a background compaction thread that runs periodically (you can
configure the scheduling for this thread). Thus, compaction happens async.

It's correct, that the current head segments is not considered for
compaction. There is also no de-duplication on write, but message will
just be appended.

You can also configure the segment size and roll behavior if you need
more "aggressive" compaction.


-Matthias

On 1/19/18 1:21 PM, Matt Farmer wrote:
> Yeah, and I thought I answered your question? I think the compaction happens 
> when new segments are created.
> 
> Sorry if I’m still misunderstanding.
> 
>> On Jan 19, 2018, at 3:55 PM, Rahul Bhattacharjee  
>> wrote:
>>
>> Thanks Matt for the response .I was asking about the log compaction
>>  of kafka topics.
>>
>> On Fri, Jan 19, 2018 at 12:36 PM, Matt Farmer  wrote:
>>
>>> Someone will need to correct me if I’m wrong, but my understanding is that
>>> a topic log on disk is divided into segments. Compaction will occur when a
>>> segment “rolls off” - so when a new active segment is created and the
>>> previous segment becomes inactive.
>>>
>>> Segments can be bounded by size and time in topic and broker configuration
>>> to get the effect that you want.
>>>
 On Jan 19, 2018, at 2:10 PM, Rahul Bhattacharjee <
>>> rahul.rec@gmail.com> wrote:

 Let's say we have a compacted topic (log.cleanup.policy=compact) where
>>> lot
 of updates happen for relatively small set of keys.
 My question is when does the compaction happen.

 In memtable , when a new update comes for an already existing key in
 memtable , the value is simple replaced.
 or,
 all the updates are associated with a offset , later the memtable is
 spilled to disk and the deletion happens during compaction phase.

 thanks,
 Rahul
>>>
>>>
> 



signature.asc
Description: OpenPGP digital signature

Re: How to always consume from latest offset in kafka-streams

[Matthias J. Sax]

That is not supported out-of-box.

Configuration "auto.offset.reset" only triggers, if there are not
committed offsets and there is KS config to change this behavior.

A possible workaround might be (but not sure if I want to recommend
this), to increase KafkaStreams commit interval via StreamsConfig (you
could set it to MAX_VALUE, effectively disable committing). Thus,
auto.offset.reset should trigger on restart. You might want to try it
out and see if it works for your... Note, if we never commit, we also
never flush KTable caches, thus you might need to disable caching, too
(by setting cache size to zero).

As an alternative, you could manipulate offsets manually before startup
using bin/kafka-consumer-groups.sh --- the application.id is the
group.id and and you could "seek to end" before you restart the application.

Hope this helps.


-Matthias

On 1/19/18 9:22 AM, Saloni Vithalani wrote:
> Our requirement is such that if a kafka-stream app is consuming a
> partition, it should start it's consumption from latest offset of that
> partition.
> 
> This seems like do-able using
> 
> streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest")
> 
> Now, let's say using above configuration, the kafka-stream app started
> consuming data from latest offset for a partition. And after some time, the
> app crashes. When the app comes back live, we want it to consume data from
> the latest offset of that partition, instead of the where it left last
> reading.
> 
> But I can't find anything that can help achieve it using kafka-streams api.
> 
> P.S. We are using kafka-1.0.0.
> 
> Saloni Vithalani
> Developer
> Email salo...@thoughtworks.com
> Telephone +91 8552889571 <8552889571>
> [image: ThoughtWorks]
> 
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

[Dong Lin]

Hey Jun,

I think we can probably have a static method in Util class to decode the
byte[]. Both KafkaConsumer implementation and the user application will be
able to decode the byte array and log its content for debug purpose. So it
seems that we can still print the information we want. It is just not
explicitly exposed in the consumer interface. Would this address the
problem here?

Yeah we can include OffsetEpoch in AdminClient. This can be added in
KIP-222? Is there something you would like me to add in this KIP?

Thanks!
Dong

On Fri, Jan 19, 2018 at 3:00 PM, Jun Rao  wrote:

> Hi, Dong,
>
> The issue with using just byte[] for OffsetEpoch is that it won't be
> printable, which makes debugging harder.
>
> Also, KIP-222 proposes a listGroupOffset() method in AdminClient. If that
> gets adopted before this KIP, we probably want to include OffsetEpoch in
> the AdminClient too.
>
> Thanks,
>
> Jun
>
>
> On Thu, Jan 18, 2018 at 6:30 PM, Dong Lin  wrote:
>
> > Hey Jun,
> >
> > I agree. I have updated the KIP to remove the class OffetEpoch and
> replace
> > OffsetEpoch with byte[] in APIs that use it. Can you see if it looks
> good?
> >
> > Thanks!
> > Dong
> >
> > On Thu, Jan 18, 2018 at 6:07 PM, Jun Rao  wrote:
> >
> > > Hi, Dong,
> > >
> > > Thanks for the updated KIP. It looks good to me now. The only thing is
> > > for OffsetEpoch.
> > > If we expose the individual fields in the class, we probably don't need
> > the
> > > encode/decode methods. If we want to hide the details of OffsetEpoch,
> we
> > > probably don't want expose the individual fields.
> > >
> > > Jun
> > >
> > > On Wed, Jan 17, 2018 at 10:10 AM, Dong Lin 
> wrote:
> > >
> > > > Thinking about point 61 more, I realize that the async zookeeper read
> > may
> > > > make it less of an issue for controller to read more zookeeper nodes.
> > > > Writing partition_epoch in the per-partition znode makes it simpler
> to
> > > > handle the broker failure between zookeeper writes for a topic
> > creation.
> > > I
> > > > have updated the KIP to use the suggested approach.
> > > >
> > > >
> > > > On Wed, Jan 17, 2018 at 9:57 AM, Dong Lin 
> wrote:
> > > >
> > > > > Hey Jun,
> > > > >
> > > > > Thanks much for the comments. Please see my comments inline.
> > > > >
> > > > > On Tue, Jan 16, 2018 at 4:38 PM, Jun Rao  wrote:
> > > > >
> > > > >> Hi, Dong,
> > > > >>
> > > > >> Thanks for the updated KIP. Looks good to me overall. Just a few
> > minor
> > > > >> comments.
> > > > >>
> > > > >> 60. OffsetAndMetadata positionAndOffsetEpoch(TopicPartition
> > > partition):
> > > > >> It
> > > > >> seems that there is no need to return metadata. We probably want
> to
> > > > return
> > > > >> sth like OffsetAndEpoch.
> > > > >>
> > > > >
> > > > > Previously I think we may want to re-use the existing class to keep
> > our
> > > > > consumer interface simpler. I have updated the KIP to add class
> > > > > OffsetAndOffsetEpoch. I didn't use OffsetAndEpoch because user may
> > > > confuse
> > > > > this name with OffsetEpoch. Does this sound OK?
> > > > >
> > > > >
> > > > >>
> > > > >> 61. Should we store partition_epoch in
> > > > >> /brokers/topics/[topic]/partitions/[partitionId] in ZK?
> > > > >>
> > > > >
> > > > > I have considered this. I think the advantage of adding the
> > > > > partition->partition_epoch map in the existing
> > > > > znode /brokers/topics/[topic]/partitions is that controller only
> > needs
> > > > to
> > > > > read one znode per topic to gets its partition_epoch information.
> > > > Otherwise
> > > > > controller may need to read one extra znode per partition to get
> the
> > > same
> > > > > information.
> > > > >
> > > > > When we delete partition or expand partition of a topic, someone
> > needs
> > > to
> > > > > modify partition->partition_epoch map in znode
> > > > > /brokers/topics/[topic]/partitions. This may seem a bit more
> > > complicated
> > > > > than simply adding or deleting znode /brokers/topics/[topic]/
> > > > partitions/[partitionId].
> > > > > But the complexity is probably similar to the existing operation of
> > > > > modifying the partition->replica_list mapping in znode
> > > > > /brokers/topics/[topic]. So not sure it is better to store the
> > > > > partition_epoch in /brokers/topics/[topic]/
> partitions/[partitionId].
> > > > What
> > > > > do you think?
> > > > >
> > > > >
> > > > >>
> > > > >> 62. For checking outdated metadata in the client, we probably want
> > to
> > > > add
> > > > >> when max_partition_epoch will be used.
> > > > >>
> > > > >
> > > > > The max_partition_epoch is used in the Proposed Changes -> Client's
> > > > > metadata refresh section to determine whether a metadata is
> outdated.
> > > And
> > > > > this formula is referenced and re-used in other sections to
> determine
> > > > > whether a metadata is outdated. Does this formula look OK?
> > > > >
> > > > >
> 

Re: [VOTE] KIP-208: Add SSL support to Kafka Connect REST interface

[Jakub Scholz]

@Jason: Actually ... wouldn't it be better to name it only
"rest.advertised.listener"?
The "security" doesn't make much sense without the protocol. What do you
think?

On Fri, Jan 19, 2018 at 10:51 PM, Jakub Scholz  wrote:

> Hi Jason,
>
> Thanks for the vote. Yeah, I think that renaming it to "
> rest.advertised.security.listener" is good idea. Unless someone else
> objects, I will update the KIP.
>
> Thanks & Regards
> Jakub
>
> On Fri, Jan 19, 2018 at 6:09 PM, Jason Gustafson 
> wrote:
>
>> +1 from me. I just had one suggestion. I saw in the discussion thread that
>> you considered allowing multiple listeners for each protocol type, but
>> ultimately rejected it. Maybe to leave the door a little more open in the
>> future for this, we could rename the config
>> "rest.advertised.security.protocol" to "rest.advertised.security.list
>> ener"
>> so that it will still make sense if we introduce a listener labeling
>> approach similar to Kafka?
>>
>> Thanks,
>> Jason
>>
>> On Fri, Jan 19, 2018 at 8:31 AM, Damian Guy  wrote:
>>
>> > Thanks Jakub!
>> >
>> > +1 (binding)
>> >
>> > On Thu, 18 Jan 2018 at 23:49 Jakub Scholz  wrote:
>> >
>> > > Hi all,
>> > >
>> > > We still need at least 2 more binding +1s. I think that the PR (
>> > > https://github.com/apache/kafka/pull/4429) is shaping good. If we get
>> > the
>> > > votes, we should be able to make the 1.1.0 release.
>> > >
>> > > Thanks & Regards
>> > > Jakub
>> > >
>> > > On Fri, Jan 5, 2018 at 4:30 AM, Ewen Cheslack-Postava <
>> e...@confluent.io
>> > >
>> > > wrote:
>> > >
>> > > > Jakub,
>> > > >
>> > > > I left a few comments in the discuss thread, but I'll also reply
>> here
>> > > just
>> > > > to bump the VOTE thread's visibility. I would like to resolve the
>> few
>> > > > comments I left, but I am effectively +1 on this, the comments I
>> left
>> > > were
>> > > > mainly details.
>> > > >
>> > > > Committers that could help with the necessary votes would probably
>> be
>> > > Gwen
>> > > > and Jason (but others more than welcome to help out too :)
>> > > >
>> > > > -Ewen
>> > > >
>> > > > On Mon, Nov 6, 2017 at 1:52 AM, Jakub Scholz 
>> wrote:
>> > > >
>> > > > > Hi all,
>> > > > >
>> > > > > Just a reminder that htis is still up for vote. I think this is
>> > > important
>> > > > > featrue which would deserve your votes.
>> > > > >
>> > > > > Regards
>> > > > > Jakub
>> > > > >
>> > > > > On Mon, Oct 30, 2017 at 9:24 PM, Jakub Scholz 
>> > wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > It seems there are no more comments for this KIP, so I would
>> like
>> > to
>> > > > > start
>> > > > > > the voting .
>> > > > > >
>> > > > > > For more details about the KIP-208 go to
>> > > > > > *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > > > 208%3A+Add+SSL+support+to+Kafka+Connect+REST+interface
>> > > > > > > > > > > 208%3A+Add+SSL+support+to+Kafka+Connect+REST+interface>*
>> > > > > >
>> > > > > > Thanks & Regards
>> > > > > > Jakub
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Re: How many kafka streams app is recommended to run on single machine in production ?

[Matthias J. Sax]

Multiple answers:

- a KafkaStreams instance start one *processing* thread by default (you
can configure more processing threads, too)

- internally, KafkaStreams uses two KafkaConsumers and one KafkaProducer
(if you turn on EOS, it uses even more KafkaProducers): a KafkaConsumer
starts a background heartbeat thread and a KafkaProducer starts a
background sender thread => you get 4 threads in total (processing, 2x
heartbeat, sender) -- if you configure two processing threads, you end
up with 8 threads in total, etc)

- there is more than one TCP connection as the consumer and the producer
(and the restore consumer, if you enable StandbyTasks) connect to the
cluster

- it's not possible to share any TPC connections atm (this would require
a mayor rewrite of consumers and producers)

- how many thread you can efficient run depends on your hardward and
workload... monitor you CPU utilization and see how buys your machine is...


Hope this helps

-Matthias

On 1/19/18 9:23 AM, Saloni Vithalani wrote:
> In our architecture, we are assuming to run three jvm processes on one
> machine (approx.) and each jvm machine can host upto 15 kafka-stream apps.
> 
> And if I am not wrong each kafka-stream app spawns one java thread. So,
> this seems like an awkward architecture to have with around 45 kafka-stream
> apps running on a single machine.
> 
> So, I have question in three parts
> 
> 1) Is my understanding correct that each kafka-stream app spawns one java
> thread ? Also, each kafka-stream starts a new tcp connection with
> kafka-broker ?
> 
> 2) Is there a way to share one tcp connection for multiple kafka-streams ?
> 
> 3) Is is difficult(not recommended) to run 45 streams on single machine ?
> The answer to this is definitely NO unless there is a real use case in
> production.
> 
> Regards,
> Saloni Vithalani
> Developer
> Email salo...@thoughtworks.com
> Telephone +91 8552889571 <8552889571>
> [image: ThoughtWorks]
> 
> 



signature.asc
Description: OpenPGP digital signature

Re: [VOTE] KIP-227: Introduce Fetch Requests that are Incremental to Increase Partition Scalability

[Jun Rao]

Hi, Colin,

Thanks for the KIP. Looks good to me overall. Just a couple of more
comments.

1. As I mentioned earlier, it might be useful to add some metrics for
monitoring the usage of the session cache. For example, it would be useful
to know how many slots are being used (or unused), # of total partitions in
the cached slots (to understand space), the eviction rate (to see if there
is any churn), etc.

2. Using max_bytes to 0 represent the removal of a partition seems
unintuitive. Perhaps it's better to either add a flag per partition or add
a removed partition list.

Jun


On Thu, Jan 18, 2018 at 6:15 PM, Colin McCabe  wrote:

> Hi all,
>
> I updated the KIP.  There is also an implementation of this KIP here:
> https://github.com/apache/kafka/pull/4418
>
> The updated implementation simplifies a few things, and adds the ability
> to incrementally add or remove individual partitions in an incremental
> fetch request.
>
> best,
> Colin
>
>
> On Tue, Dec 19, 2017, at 19:28, Colin McCabe wrote:
> > Hi all,
> >
> > I'd like to start the vote on KIP-227: Incremental Fetch Requests.
> >
> > The KIP is here:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 227%3A+Introduce+Incremental+FetchRequests+to+Increase+
> Partition+Scalability
> >
> > and discussion thread earlier:
> > https://www.mail-archive.com/dev@kafka.apache.org/msg83115.html
> >
> > thanks,
> > Colin
>

Re: [DISCUSS] KIP-208: Add SSL support to Kafka Connect REST interface

[Jakub Scholz]

FYI: For those not following the VOTE thread  I updated the KIP and
changed the field "rest.advertised.security.protocol" to
"rest.advertised.security.listener" as suggested by Jason.

On Fri, Jan 19, 2018 at 11:29 AM, Jakub Scholz  wrote:

> Hi all,
>
> I did one more update to the KIP-208. I added the
> "ssl.endpoint.identification.algorithm" to the list of supported options.
> It can be used to enable / disable the hostname validation when the Kafka
> Connect nodes are forwarding the requests to the leader. It is minor but
> useful change, so I hope nobody minds :-).
>
> Thanks & Regards
> Jakub
>
> On Fri, Jan 19, 2018 at 12:53 AM, Jakub Scholz  wrote:
>
>> Hi Randall,
>>
>> Yes the KIP should be up to date. The VOTE thread is actually running
>> already for more than 2 months. So the only thing we need is the votes. I
>> pinged the vote thread so that it gets more attention.
>>
>> Thanks & Regards
>> Jakub
>>
>> On Thu, Jan 18, 2018 at 7:33 PM, Randall Hauch  wrote:
>>
>>> Jakub, have you had a chance to update the KIP with the latest changes?
>>> Would be great to start the vote today so that it's open long enough to
>>> adopt before the deadline on Tuesday. Let me know if I can help.
>>>
>>> On Wed, Jan 17, 2018 at 1:25 AM, Jakub Scholz  wrote:
>>>
>>> > I have been thinking about this a bit more yesterday while updating the
>>> > code. I think you are right, we should use only the prefixed values if
>>> at
>>> > least one of them exists. Even I got quite easily confused what setup
>>> is
>>> > actually used when the fields are mixed :-). Randall was also in
>>> favour of
>>> > this approach. So I think we should go this way. I will update the KIP
>>> > accordingly.
>>> >
>>> >
>>> > > I'm fine with consistency, but maybe the thing to do here then is to
>>> > ensure
>>> > > that we definitely log the "effective" or "derived" config before
>>> using
>>> > it
>>> > > so there is at least some useful trace of what we actually used that
>>> can
>>> > be
>>> > > helpful in debugging.
>>> >
>>>
>>
>>
>

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

[Jun Rao]

Hi, Dong,

The issue with using just byte[] for OffsetEpoch is that it won't be
printable, which makes debugging harder.

Also, KIP-222 proposes a listGroupOffset() method in AdminClient. If that
gets adopted before this KIP, we probably want to include OffsetEpoch in
the AdminClient too.

Thanks,

Jun


On Thu, Jan 18, 2018 at 6:30 PM, Dong Lin  wrote:

> Hey Jun,
>
> I agree. I have updated the KIP to remove the class OffetEpoch and replace
> OffsetEpoch with byte[] in APIs that use it. Can you see if it looks good?
>
> Thanks!
> Dong
>
> On Thu, Jan 18, 2018 at 6:07 PM, Jun Rao  wrote:
>
> > Hi, Dong,
> >
> > Thanks for the updated KIP. It looks good to me now. The only thing is
> > for OffsetEpoch.
> > If we expose the individual fields in the class, we probably don't need
> the
> > encode/decode methods. If we want to hide the details of OffsetEpoch, we
> > probably don't want expose the individual fields.
> >
> > Jun
> >
> > On Wed, Jan 17, 2018 at 10:10 AM, Dong Lin  wrote:
> >
> > > Thinking about point 61 more, I realize that the async zookeeper read
> may
> > > make it less of an issue for controller to read more zookeeper nodes.
> > > Writing partition_epoch in the per-partition znode makes it simpler to
> > > handle the broker failure between zookeeper writes for a topic
> creation.
> > I
> > > have updated the KIP to use the suggested approach.
> > >
> > >
> > > On Wed, Jan 17, 2018 at 9:57 AM, Dong Lin  wrote:
> > >
> > > > Hey Jun,
> > > >
> > > > Thanks much for the comments. Please see my comments inline.
> > > >
> > > > On Tue, Jan 16, 2018 at 4:38 PM, Jun Rao  wrote:
> > > >
> > > >> Hi, Dong,
> > > >>
> > > >> Thanks for the updated KIP. Looks good to me overall. Just a few
> minor
> > > >> comments.
> > > >>
> > > >> 60. OffsetAndMetadata positionAndOffsetEpoch(TopicPartition
> > partition):
> > > >> It
> > > >> seems that there is no need to return metadata. We probably want to
> > > return
> > > >> sth like OffsetAndEpoch.
> > > >>
> > > >
> > > > Previously I think we may want to re-use the existing class to keep
> our
> > > > consumer interface simpler. I have updated the KIP to add class
> > > > OffsetAndOffsetEpoch. I didn't use OffsetAndEpoch because user may
> > > confuse
> > > > this name with OffsetEpoch. Does this sound OK?
> > > >
> > > >
> > > >>
> > > >> 61. Should we store partition_epoch in
> > > >> /brokers/topics/[topic]/partitions/[partitionId] in ZK?
> > > >>
> > > >
> > > > I have considered this. I think the advantage of adding the
> > > > partition->partition_epoch map in the existing
> > > > znode /brokers/topics/[topic]/partitions is that controller only
> needs
> > > to
> > > > read one znode per topic to gets its partition_epoch information.
> > > Otherwise
> > > > controller may need to read one extra znode per partition to get the
> > same
> > > > information.
> > > >
> > > > When we delete partition or expand partition of a topic, someone
> needs
> > to
> > > > modify partition->partition_epoch map in znode
> > > > /brokers/topics/[topic]/partitions. This may seem a bit more
> > complicated
> > > > than simply adding or deleting znode /brokers/topics/[topic]/
> > > partitions/[partitionId].
> > > > But the complexity is probably similar to the existing operation of
> > > > modifying the partition->replica_list mapping in znode
> > > > /brokers/topics/[topic]. So not sure it is better to store the
> > > > partition_epoch in /brokers/topics/[topic]/partitions/[partitionId].
> > > What
> > > > do you think?
> > > >
> > > >
> > > >>
> > > >> 62. For checking outdated metadata in the client, we probably want
> to
> > > add
> > > >> when max_partition_epoch will be used.
> > > >>
> > > >
> > > > The max_partition_epoch is used in the Proposed Changes -> Client's
> > > > metadata refresh section to determine whether a metadata is outdated.
> > And
> > > > this formula is referenced and re-used in other sections to determine
> > > > whether a metadata is outdated. Does this formula look OK?
> > > >
> > > >
> > > >>
> > > >> 63. "The leader_epoch should be the largest leader_epoch of messages
> > > whose
> > > >> offset < the commit offset. If no message has been consumed since
> > > consumer
> > > >> initialization, the leader_epoch from seek(...) or
> OffsetFetchResponse
> > > >> should be used. The partition_epoch should be read from the last
> > > >> FetchResponse corresponding to the given partition and commit
> offset.
> > ":
> > > >> leader_epoch and partition_epoch are associated with an offset. So,
> if
> > > no
> > > >> message is consumed, there is no offset and therefore there is no
> need
> > > to
> > > >> read leader_epoch and partition_epoch. Also, the leader_epoch
> > associated
> > > >> with the offset should just come from the messages returned in the
> > fetch
> > > >> response.
> > > >>
> > > >
> > > > I am thinking that, if user calls seek(..) 

Re: [VOTE] KIP-222 - Add "describe consumer group" to KafkaAdminClient

[Jun Rao]

Hi, Jorge,

Thanks for the KIP. Looks good to me overall. A few comments below.

1. It seems that ConsumerDescription should be MemberDescription?

2. Each offset can have an optional metadata. So, in
ListGroupOffsetsResult, perhaps it's better to have
KafkaFuture>, where
OffsetAndMetadata contains an offset and a metadata of String.

3. As Jason mentioned in the discussion, it would be nice to extend this
api to support general group management, instead of just the consumer group
in the future. For that, it might be better for MemberDescription to have
assignment of type Assignment, which consists of a list of partitions.
Then, in the future, we can add other fields to Assignment.

Jun


On Thu, Jan 18, 2018 at 9:45 AM, Mickael Maison 
wrote:

> +1 (non binding), thanks
>
> On Thu, Jan 18, 2018 at 5:41 PM, Colin McCabe  wrote:
> > +1 (non-binding)
> >
> > Colin
> >
> >
> > On Thu, Jan 18, 2018, at 07:36, Ted Yu wrote:
> >> +1
> >>  Original message From: Bill Bejeck 
> >> Date: 1/18/18  6:59 AM  (GMT-08:00) To: dev@kafka.apache.org Subject:
> >> Re: [VOTE] KIP-222 - Add "describe consumer group" to KafkaAdminClient
> >> Thanks for the KIP
> >>
> >> +1
> >>
> >> Bill
> >>
> >> On Thu, Jan 18, 2018 at 4:24 AM, Rajini Sivaram <
> rajinisiva...@gmail.com>
> >> wrote:
> >>
> >> > +1 (binding)
> >> >
> >> > Thanks for the KIP, Jorge.
> >> >
> >> > Regards,
> >> >
> >> > Rajini
> >> >
> >> > On Wed, Jan 17, 2018 at 9:04 PM, Guozhang Wang 
> wrote:
> >> >
> >> > > +1 (binding). Thanks Jorge.
> >> > >
> >> > >
> >> > > Guozhang
> >> > >
> >> > > On Wed, Jan 17, 2018 at 11:29 AM, Gwen Shapira 
> >> > wrote:
> >> > >
> >> > > > Hey, since there were no additional comments in the discussion,
> I'd
> >> > like
> >> > > to
> >> > > > resume the voting.
> >> > > >
> >> > > > +1 (binding)
> >> > > >
> >> > > > On Fri, Nov 17, 2017 at 9:15 AM Guozhang Wang  >
> >> > > wrote:
> >> > > >
> >> > > > > Hello Jorge,
> >> > > > >
> >> > > > > I left some comments on the discuss thread. The wiki page itself
> >> > looks
> >> > > > good
> >> > > > > overall.
> >> > > > >
> >> > > > >
> >> > > > > Guozhang
> >> > > > >
> >> > > > > On Tue, Nov 14, 2017 at 10:02 AM, Jorge Esteban Quilcate Otoya <
> >> > > > > quilcate.jo...@gmail.com> wrote:
> >> > > > >
> >> > > > > > Added.
> >> > > > > >
> >> > > > > > El mar., 14 nov. 2017 a las 19:00, Ted Yu (<
> yuzhih...@gmail.com>)
> >> > > > > > escribió:
> >> > > > > >
> >> > > > > > > Please fill in JIRA number in Status section.
> >> > > > > > >
> >> > > > > > > On Tue, Nov 14, 2017 at 9:57 AM, Jorge Esteban Quilcate
> Otoya <
> >> > > > > > > quilcate.jo...@gmail.com> wrote:
> >> > > > > > >
> >> > > > > > > > JIRA issue title updated.
> >> > > > > > > >
> >> > > > > > > > El mar., 14 nov. 2017 a las 18:45, Ted Yu (<
> >> > yuzhih...@gmail.com
> >> > > >)
> >> > > > > > > > escribió:
> >> > > > > > > >
> >> > > > > > > > > Can you fill in JIRA number (KAFKA-6058
> >> > > > > > > > > ) ?
> >> > > > > > > > >
> >> > > > > > > > > If one JIRA is used for the two additions, consider
> updating
> >> > > the
> >> > > > > JIRA
> >> > > > > > > > > title.
> >> > > > > > > > >
> >> > > > > > > > > On Tue, Nov 14, 2017 at 9:04 AM, Jorge Esteban Quilcate
> >> > Otoya <
> >> > > > > > > > > quilcate.jo...@gmail.com> wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hi all,
> >> > > > > > > > > >
> >> > > > > > > > > > As I didn't see any further discussion around this
> KIP, I'd
> >> > > > like
> >> > > > > to
> >> > > > > > > > start
> >> > > > > > > > > > voting.
> >> > > > > > > > > >
> >> > > > > > > > > > KIP documentation:
> >> > > > > > > > > >
> >> > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> >> > > > > > > > action?pageId=74686265
> >> > > > > > > > > >
> >> > > > > > > > > > Cheers,
> >> > > > > > > > > > Jorge.
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > -- Guozhang
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > -- Guozhang
> >> > >
> >> >
>

Re: [VOTE] KIP-208: Add SSL support to Kafka Connect REST interface

[Jakub Scholz]

Hi Jason,

Thanks for the vote. Yeah, I think that renaming it to "
rest.advertised.security.listener" is good idea. Unless someone else
objects, I will update the KIP.

Thanks & Regards
Jakub

On Fri, Jan 19, 2018 at 6:09 PM, Jason Gustafson  wrote:

> +1 from me. I just had one suggestion. I saw in the discussion thread that
> you considered allowing multiple listeners for each protocol type, but
> ultimately rejected it. Maybe to leave the door a little more open in the
> future for this, we could rename the config
> "rest.advertised.security.protocol" to "rest.advertised.security.listener"
> so that it will still make sense if we introduce a listener labeling
> approach similar to Kafka?
>
> Thanks,
> Jason
>
> On Fri, Jan 19, 2018 at 8:31 AM, Damian Guy  wrote:
>
> > Thanks Jakub!
> >
> > +1 (binding)
> >
> > On Thu, 18 Jan 2018 at 23:49 Jakub Scholz  wrote:
> >
> > > Hi all,
> > >
> > > We still need at least 2 more binding +1s. I think that the PR (
> > > https://github.com/apache/kafka/pull/4429) is shaping good. If we get
> > the
> > > votes, we should be able to make the 1.1.0 release.
> > >
> > > Thanks & Regards
> > > Jakub
> > >
> > > On Fri, Jan 5, 2018 at 4:30 AM, Ewen Cheslack-Postava <
> e...@confluent.io
> > >
> > > wrote:
> > >
> > > > Jakub,
> > > >
> > > > I left a few comments in the discuss thread, but I'll also reply here
> > > just
> > > > to bump the VOTE thread's visibility. I would like to resolve the few
> > > > comments I left, but I am effectively +1 on this, the comments I left
> > > were
> > > > mainly details.
> > > >
> > > > Committers that could help with the necessary votes would probably be
> > > Gwen
> > > > and Jason (but others more than welcome to help out too :)
> > > >
> > > > -Ewen
> > > >
> > > > On Mon, Nov 6, 2017 at 1:52 AM, Jakub Scholz 
> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Just a reminder that htis is still up for vote. I think this is
> > > important
> > > > > featrue which would deserve your votes.
> > > > >
> > > > > Regards
> > > > > Jakub
> > > > >
> > > > > On Mon, Oct 30, 2017 at 9:24 PM, Jakub Scholz 
> > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > It seems there are no more comments for this KIP, so I would like
> > to
> > > > > start
> > > > > > the voting .
> > > > > >
> > > > > > For more details about the KIP-208 go to
> > > > > > *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 208%3A+Add+SSL+support+to+Kafka+Connect+REST+interface
> > > > > >  > > > > 208%3A+Add+SSL+support+to+Kafka+Connect+REST+interface>*
> > > > > >
> > > > > > Thanks & Regards
> > > > > > Jakub
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Kafka compacted topic question.

[Matt Farmer]

Yeah, and I thought I answered your question? I think the compaction happens 
when new segments are created.

Sorry if I’m still misunderstanding.

> On Jan 19, 2018, at 3:55 PM, Rahul Bhattacharjee  
> wrote:
> 
> Thanks Matt for the response .I was asking about the log compaction
>  of kafka topics.
> 
> On Fri, Jan 19, 2018 at 12:36 PM, Matt Farmer  wrote:
> 
>> Someone will need to correct me if I’m wrong, but my understanding is that
>> a topic log on disk is divided into segments. Compaction will occur when a
>> segment “rolls off” - so when a new active segment is created and the
>> previous segment becomes inactive.
>> 
>> Segments can be bounded by size and time in topic and broker configuration
>> to get the effect that you want.
>> 
>>> On Jan 19, 2018, at 2:10 PM, Rahul Bhattacharjee <
>> rahul.rec@gmail.com> wrote:
>>> 
>>> Let's say we have a compacted topic (log.cleanup.policy=compact) where
>> lot
>>> of updates happen for relatively small set of keys.
>>> My question is when does the compaction happen.
>>> 
>>> In memtable , when a new update comes for an already existing key in
>>> memtable , the value is simple replaced.
>>> or,
>>> all the updates are associated with a offset , later the memtable is
>>> spilled to disk and the deletion happens during compaction phase.
>>> 
>>> thanks,
>>> Rahul
>> 
>> 

Re: [VOTE] KIP-208: Add SSL support to Kafka Connect REST interface

[Gwen Shapira]

+1 (binding), Thank you!!!

On Fri, Jan 19, 2018 at 9:09 AM Jason Gustafson  wrote:

> +1 from me. I just had one suggestion. I saw in the discussion thread that
> you considered allowing multiple listeners for each protocol type, but
> ultimately rejected it. Maybe to leave the door a little more open in the
> future for this, we could rename the config
> "rest.advertised.security.protocol" to "rest.advertised.security.listener"
> so that it will still make sense if we introduce a listener labeling
> approach similar to Kafka?
>
> Thanks,
> Jason
>
> On Fri, Jan 19, 2018 at 8:31 AM, Damian Guy  wrote:
>
> > Thanks Jakub!
> >
> > +1 (binding)
> >
> > On Thu, 18 Jan 2018 at 23:49 Jakub Scholz  wrote:
> >
> > > Hi all,
> > >
> > > We still need at least 2 more binding +1s. I think that the PR (
> > > https://github.com/apache/kafka/pull/4429) is shaping good. If we get
> > the
> > > votes, we should be able to make the 1.1.0 release.
> > >
> > > Thanks & Regards
> > > Jakub
> > >
> > > On Fri, Jan 5, 2018 at 4:30 AM, Ewen Cheslack-Postava <
> e...@confluent.io
> > >
> > > wrote:
> > >
> > > > Jakub,
> > > >
> > > > I left a few comments in the discuss thread, but I'll also reply here
> > > just
> > > > to bump the VOTE thread's visibility. I would like to resolve the few
> > > > comments I left, but I am effectively +1 on this, the comments I left
> > > were
> > > > mainly details.
> > > >
> > > > Committers that could help with the necessary votes would probably be
> > > Gwen
> > > > and Jason (but others more than welcome to help out too :)
> > > >
> > > > -Ewen
> > > >
> > > > On Mon, Nov 6, 2017 at 1:52 AM, Jakub Scholz 
> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Just a reminder that htis is still up for vote. I think this is
> > > important
> > > > > featrue which would deserve your votes.
> > > > >
> > > > > Regards
> > > > > Jakub
> > > > >
> > > > > On Mon, Oct 30, 2017 at 9:24 PM, Jakub Scholz 
> > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > It seems there are no more comments for this KIP, so I would like
> > to
> > > > > start
> > > > > > the voting .
> > > > > >
> > > > > > For more details about the KIP-208 go to
> > > > > > *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 208%3A+Add+SSL+support+to+Kafka+Connect+REST+interface
> > > > > >  > > > > 208%3A+Add+SSL+support+to+Kafka+Connect+REST+interface>*
> > > > > >
> > > > > > Thanks & Regards
> > > > > > Jakub
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Kafka compacted topic question.

[Rahul Bhattacharjee]

Thanks Matt for the response .I was asking about the log compaction
 of kafka topics.

On Fri, Jan 19, 2018 at 12:36 PM, Matt Farmer  wrote:

> Someone will need to correct me if I’m wrong, but my understanding is that
> a topic log on disk is divided into segments. Compaction will occur when a
> segment “rolls off” - so when a new active segment is created and the
> previous segment becomes inactive.
>
> Segments can be bounded by size and time in topic and broker configuration
> to get the effect that you want.
>
> > On Jan 19, 2018, at 2:10 PM, Rahul Bhattacharjee <
> rahul.rec@gmail.com> wrote:
> >
> > Let's say we have a compacted topic (log.cleanup.policy=compact) where
> lot
> > of updates happen for relatively small set of keys.
> > My question is when does the compaction happen.
> >
> > In memtable , when a new update comes for an already existing key in
> > memtable , the value is simple replaced.
> > or,
> > all the updates are associated with a offset , later the memtable is
> > spilled to disk and the deletion happens during compaction phase.
> >
> > thanks,
> > Rahul
>
>

Re: Kafka compacted topic question.

[Matt Farmer]

Someone will need to correct me if I’m wrong, but my understanding is that a 
topic log on disk is divided into segments. Compaction will occur when a 
segment “rolls off” - so when a new active segment is created and the previous 
segment becomes inactive.

Segments can be bounded by size and time in topic and broker configuration to 
get the effect that you want.

> On Jan 19, 2018, at 2:10 PM, Rahul Bhattacharjee  
> wrote:
> 
> Let's say we have a compacted topic (log.cleanup.policy=compact) where lot
> of updates happen for relatively small set of keys.
> My question is when does the compaction happen.
> 
> In memtable , when a new update comes for an already existing key in
> memtable , the value is simple replaced.
> or,
> all the updates are associated with a offset , later the memtable is
> spilled to disk and the deletion happens during compaction phase.
> 
> thanks,
> Rahul

Re: How to always consume from latest offset in kafka-streams

[Matt Farmer]

That config setting will only work if there are no offsets stored in the 
consumer offsets target.

Something I’ve done in the past is to make the application.id config setting 
have a random string component to it. So have “my-app-name-[randomchars]” or 
some such. This ensures that there are never pre-existing offsets. Would that 
be an acceptable method for you?

Matt

> On Jan 19, 2018, at 12:22 PM, Saloni Vithalani  
> wrote:
> 
> Our requirement is such that if a kafka-stream app is consuming a
> partition, it should start it's consumption from latest offset of that
> partition.
> 
> This seems like do-able using
> 
> streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest")
> 
> Now, let's say using above configuration, the kafka-stream app started
> consuming data from latest offset for a partition. And after some time, the
> app crashes. When the app comes back live, we want it to consume data from
> the latest offset of that partition, instead of the where it left last
> reading.
> 
> But I can't find anything that can help achieve it using kafka-streams api.
> 
> P.S. We are using kafka-1.0.0.
> 
> Saloni Vithalani
> Developer
> Email salo...@thoughtworks.com
> Telephone +91 8552889571 <8552889571>
> [image: ThoughtWorks]
> 

Kafka compacted topic question.

[Rahul Bhattacharjee]

Let's say we have a compacted topic (log.cleanup.policy=compact) where lot
of updates happen for relatively small set of keys.
My question is when does the compaction happen.

In memtable , when a new update comes for an already existing key in
memtable , the value is simple replaced.
or,
all the updates are associated with a offset , later the memtable is
spilled to disk and the deletion happens during compaction phase.

thanks,
Rahul

How to always consume from latest offset in kafka-streams

[Saloni Vithalani]

Our requirement is such that if a kafka-stream app is consuming a
partition, it should start it's consumption from latest offset of that
partition.

This seems like do-able using

streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest")

Now, let's say using above configuration, the kafka-stream app started
consuming data from latest offset for a partition. And after some time, the
app crashes. When the app comes back live, we want it to consume data from
the latest offset of that partition, instead of the where it left last
reading.

But I can't find anything that can help achieve it using kafka-streams api.

P.S. We are using kafka-1.0.0.

Saloni Vithalani
Developer
Email salo...@thoughtworks.com
Telephone +91 8552889571 <8552889571>
[image: ThoughtWorks]

How many kafka streams app is recommended to run on single machine in production ?

[Saloni Vithalani]

In our architecture, we are assuming to run three jvm processes on one
machine (approx.) and each jvm machine can host upto 15 kafka-stream apps.

And if I am not wrong each kafka-stream app spawns one java thread. So,
this seems like an awkward architecture to have with around 45 kafka-stream
apps running on a single machine.

So, I have question in three parts

1) Is my understanding correct that each kafka-stream app spawns one java
thread ? Also, each kafka-stream starts a new tcp connection with
kafka-broker ?

2) Is there a way to share one tcp connection for multiple kafka-streams ?

3) Is is difficult(not recommended) to run 45 streams on single machine ?
The answer to this is definitely NO unless there is a real use case in
production.

Regards,
Saloni Vithalani
Developer
Email salo...@thoughtworks.com
Telephone +91 8552889571 <8552889571>
[image: ThoughtWorks]

Re: [VOTE] KIP-208: Add SSL support to Kafka Connect REST interface

[Jason Gustafson]

+1 from me. I just had one suggestion. I saw in the discussion thread that
you considered allowing multiple listeners for each protocol type, but
ultimately rejected it. Maybe to leave the door a little more open in the
future for this, we could rename the config
"rest.advertised.security.protocol" to "rest.advertised.security.listener"
so that it will still make sense if we introduce a listener labeling
approach similar to Kafka?

Thanks,
Jason

On Fri, Jan 19, 2018 at 8:31 AM, Damian Guy  wrote:

> Thanks Jakub!
>
> +1 (binding)
>
> On Thu, 18 Jan 2018 at 23:49 Jakub Scholz  wrote:
>
> > Hi all,
> >
> > We still need at least 2 more binding +1s. I think that the PR (
> > https://github.com/apache/kafka/pull/4429) is shaping good. If we get
> the
> > votes, we should be able to make the 1.1.0 release.
> >
> > Thanks & Regards
> > Jakub
> >
> > On Fri, Jan 5, 2018 at 4:30 AM, Ewen Cheslack-Postava  >
> > wrote:
> >
> > > Jakub,
> > >
> > > I left a few comments in the discuss thread, but I'll also reply here
> > just
> > > to bump the VOTE thread's visibility. I would like to resolve the few
> > > comments I left, but I am effectively +1 on this, the comments I left
> > were
> > > mainly details.
> > >
> > > Committers that could help with the necessary votes would probably be
> > Gwen
> > > and Jason (but others more than welcome to help out too :)
> > >
> > > -Ewen
> > >
> > > On Mon, Nov 6, 2017 at 1:52 AM, Jakub Scholz  wrote:
> > >
> > > > Hi all,
> > > >
> > > > Just a reminder that htis is still up for vote. I think this is
> > important
> > > > featrue which would deserve your votes.
> > > >
> > > > Regards
> > > > Jakub
> > > >
> > > > On Mon, Oct 30, 2017 at 9:24 PM, Jakub Scholz 
> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > It seems there are no more comments for this KIP, so I would like
> to
> > > > start
> > > > > the voting .
> > > > >
> > > > > For more details about the KIP-208 go to
> > > > > *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 208%3A+Add+SSL+support+to+Kafka+Connect+REST+interface
> > > > >  > > > 208%3A+Add+SSL+support+to+Kafka+Connect+REST+interface>*
> > > > >
> > > > > Thanks & Regards
> > > > > Jakub
> > > > >
> > > >
> > >
> >
>

Build failed in Jenkins: kafka-0.10.2-jdk7 #201

[Apache Jenkins Server]

See 


Changes:

[rajinisivaram] MINOR: additional check to follower fetch handling (#4448)

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H28 (ubuntu xenial) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/0.10.2^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/0.10.2^{commit} # timeout=10
Checking out Revision d2932ad370c5b56edac9d99e6d75f199537a569f 
(refs/remotes/origin/0.10.2)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d2932ad370c5b56edac9d99e6d75f199537a569f
Commit message: "MINOR: additional check to follower fetch handling (#4448)"
 > git rev-list 6a0b84aa0f2bf43eff7293fb285043e9584f32c3 # timeout=10
ERROR: No tool found matching GRADLE_2_4_RC_2_HOME
[kafka-0.10.2-jdk7] $ /bin/bash -xe /tmp/jenkins8509646240663339132.sh
+ rm -rf 
+ /bin/gradle
/tmp/jenkins8509646240663339132.sh: line 4: /bin/gradle: No such file or 
directory
Build step 'Execute shell' marked build as failure
Recording test results
ERROR: No tool found matching GRADLE_2_4_RC_2_HOME
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
ERROR: No tool found matching GRADLE_2_4_RC_2_HOME
Not sending mail to unregistered user rajinisiva...@googlemail.com

Re: [VOTE] KIP-208: Add SSL support to Kafka Connect REST interface

[Damian Guy]

Thanks Jakub!

+1 (binding)

On Thu, 18 Jan 2018 at 23:49 Jakub Scholz  wrote:

> Hi all,
>
> We still need at least 2 more binding +1s. I think that the PR (
> https://github.com/apache/kafka/pull/4429) is shaping good. If we get the
> votes, we should be able to make the 1.1.0 release.
>
> Thanks & Regards
> Jakub
>
> On Fri, Jan 5, 2018 at 4:30 AM, Ewen Cheslack-Postava 
> wrote:
>
> > Jakub,
> >
> > I left a few comments in the discuss thread, but I'll also reply here
> just
> > to bump the VOTE thread's visibility. I would like to resolve the few
> > comments I left, but I am effectively +1 on this, the comments I left
> were
> > mainly details.
> >
> > Committers that could help with the necessary votes would probably be
> Gwen
> > and Jason (but others more than welcome to help out too :)
> >
> > -Ewen
> >
> > On Mon, Nov 6, 2017 at 1:52 AM, Jakub Scholz  wrote:
> >
> > > Hi all,
> > >
> > > Just a reminder that htis is still up for vote. I think this is
> important
> > > featrue which would deserve your votes.
> > >
> > > Regards
> > > Jakub
> > >
> > > On Mon, Oct 30, 2017 at 9:24 PM, Jakub Scholz  wrote:
> > >
> > > > Hi,
> > > >
> > > > It seems there are no more comments for this KIP, so I would like to
> > > start
> > > > the voting .
> > > >
> > > > For more details about the KIP-208 go to
> > > > *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 208%3A+Add+SSL+support+to+Kafka+Connect+REST+interface
> > > >  > > 208%3A+Add+SSL+support+to+Kafka+Connect+REST+interface>*
> > > >
> > > > Thanks & Regards
> > > > Jakub
> > > >
> > >
> >
>

[VOTE] KIP-248: Create New ConfigCommand That Uses The New AdminClient

[Viktor Somogyi]

Hi all,

I'd like to start the vote on KIP-248: Create New ConfigCommand That Uses
The New AdminClient.

The KIP can be read here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-248+-+Create+New+ConfigCommand+That+Uses+The+New+AdminClient

The discussion thread is at
https://lists.apache.org/thread.html/6c0b347a52b51f618d7b86b44d0dce65a6079764358fef619d4807c9@%3Cdev.kafka.apache.org%3E
.

Thank you,
Viktor

Re: [DISCUSS] KIP-248 Create New ConfigCommand That Uses The New AdminClient

[Viktor Somogyi]

Hi Rajini,

Ok, I think I got you. I wasn't calculating with the fact that the parent
might not be set, therefore it could be a default user as well or even the
default client if nothing else is set (supposing we're talking about the
 example). So if I'm correct, the quota will be applied in
the order of the above points. In this case your argument is absolutely
valid. I'll modify the QuotaSource.

About your last point: yes, I was hesitating a lot. I thought the interface
would be simpler but after removing the helpers it's not that scary
afterall :).
I'll start the vote.

Viktor


On Thu, Jan 18, 2018 at 7:59 PM, Rajini Sivaram 
wrote:

> Hi Viktor,
>
> Thanks for the updates.
>
> *QuotaSource* currently has *Self/Default/Parent*. Not sure if that is
> sufficient.
> For the entity , quota could be used from any of these
> configs:
>
>1. /config/users//clients/
>2. /config/users//clients/
>3. /config/users/
>4. /config/users//clients/
>5. /config/users//clients/
>6. /config/users/
>7. /config/clients/
>8. /config/clients/
>
> So perhaps we should have a *QuotaSource* entry for each of these eight?
>
> A couple of minor points:
>
>- *Help Message* still uses --config.properties
>- The other AdminClient APIs don't use aliases for various collections.
>So not sure if we need the aliases here. I think you can leave it as-is
> and
>see what others think.
>
> Yes, please do start the voting thread to make it in time for the KIP
> freeze.
>
> Thank you,
>
> Rajini
>
>
> On Thu, Jan 18, 2018 at 6:15 PM, Viktor Somogyi 
> wrote:
>
> > Rajini, I have updated the KIP as agreed. Could you please have a second
> > look at it?
> > I have also added a section about SCRAM:
> > "To enable describing and altering SCRAM credentials we will use the
> > DescribeConfigs and AlterConfigs protocols. There are no changes in the
> > protocol's structure but we will allow the USER resource type to be
> passed
> > in the protocol. When this happens, the server will know that SCRAM
> configs
> > are asked and will send them in the response.  In case of AlterConfigs
> if a
> > USER resource type is passed it will validate if there are only SCRAM
> > credentials are changed. If not, then will fail with
> > InvalidRequestException
> > ."
> >
> > If you don't have any comments, we might start voting as we're close to
> KIP
> > freeze.
> >
> > On Thu, Jan 18, 2018 at 12:12 PM, Viktor Somogyi <
> viktorsomo...@gmail.com>
> > wrote:
> >
> > > 3. Ok, I'll remove this from the KIP for now and perhaps add a future
> > > considerations section with the idea.
> > >
> > > 9. Ok, I can do that.
> > >
> > > On Thu, Jan 18, 2018 at 11:18 AM, Rajini Sivaram <
> > rajinisiva...@gmail.com>
> > > wrote:
> > >
> > >> Hi Viktor,
> > >>
> > >> 3. Agree that it would be better to use something like
> ConfigEntityList
> > >> rather than ListQuotas. But I would leave it out for now since we are
> so
> > >> close to KIP freeze. We can introduce it later if required. Earlier, I
> > was
> > >> thinking that if we just wanted to get a list of entities without
> their
> > >> actual quota values, you could have an option in DescribeQuotas to
> > return
> > >> the entities without the quota values. But actually that doesn't make
> > >> sense
> > >> since you will need to read all the ZK entries and find the ones with
> > >> quotas in the first place. So let's just leave DescribeQuotas as-is.
> > >>
> > >> 7. Yes, with client-id quotas at the lowest level. The full list in
> the
> > >> order of precedence is here:
> > >> https://kafka.apache.org/documentation/#design_quotasconfig
> > >>
> > >> 9. One more suggestion. Since DescribeQuotas and AlterQuotas are
> > specific
> > >> to quotas, we could use *quota* instead of *config* in the protocol
> (and
> > >> AdminClient API). Instead of *config_name*, we could use a
> *quota_type*
> > >> enum (we have three types). And *config_value *could be *quota_value
> > *that
> > >> is a double rather than a string*,*
> > >>
> > >> On Thu, Jan 18, 2018 at 9:38 AM, Viktor Somogyi <
> > viktorsomo...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi Rajini,
> > >> >
> > >> > 1. Yes, --adminclient.config it is. I missed that, sorry :)
> > >> >
> > >> > 3. Indeed SCRAM in this case can raise complications. Since we'd
> like
> > to
> > >> > handle altering SCRAM credentials via AlterConfigs, I think we
> should
> > >> use
> > >> > DescribeConfigs to describe them. That is, to describe all the
> > entities
> > >> of
> > >> > a given type we might need to introduce some kind of generic way of
> > >> getting
> > >> > metadata of config entities instead of ListQuotas which is very
> > >> specific.
> > >> > Therefore instead of ListQuotas we could have a ConfigMetadata (or
> > >> > ConfigEntityList) protocol tailored to the needs of the admin
> client.
> > >> This
> > >> > protocol would accept a list of config 

[jira] [Created] (KAFKA-6463) Review logging level for user errors in AdminManager

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-6463:
-

 Summary: Review logging level for user errors in AdminManager
 Key: KAFKA-6463
 URL: https://issues.apache.org/jira/browse/KAFKA-6463
 Project: Kafka
  Issue Type: Improvement
Reporter: Rajini Sivaram
 Fix For: 1.1.0


AdminManager currently logs errors due to bad requests at INFO level (e.g. 
alter configs with bad value). In other components, I think we only log user 
errors are either not logged or logged at a lower logging level. We should 
review logging in AdminManager.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-208: Add SSL support to Kafka Connect REST interface

[Jakub Scholz]

Hi all,

I did one more update to the KIP-208. I added the
"ssl.endpoint.identification.algorithm" to the list of supported options.
It can be used to enable / disable the hostname validation when the Kafka
Connect nodes are forwarding the requests to the leader. It is minor but
useful change, so I hope nobody minds :-).

Thanks & Regards
Jakub

On Fri, Jan 19, 2018 at 12:53 AM, Jakub Scholz  wrote:

> Hi Randall,
>
> Yes the KIP should be up to date. The VOTE thread is actually running
> already for more than 2 months. So the only thing we need is the votes. I
> pinged the vote thread so that it gets more attention.
>
> Thanks & Regards
> Jakub
>
> On Thu, Jan 18, 2018 at 7:33 PM, Randall Hauch  wrote:
>
>> Jakub, have you had a chance to update the KIP with the latest changes?
>> Would be great to start the vote today so that it's open long enough to
>> adopt before the deadline on Tuesday. Let me know if I can help.
>>
>> On Wed, Jan 17, 2018 at 1:25 AM, Jakub Scholz  wrote:
>>
>> > I have been thinking about this a bit more yesterday while updating the
>> > code. I think you are right, we should use only the prefixed values if
>> at
>> > least one of them exists. Even I got quite easily confused what setup is
>> > actually used when the fields are mixed :-). Randall was also in favour
>> of
>> > this approach. So I think we should go this way. I will update the KIP
>> > accordingly.
>> >
>> >
>> > > I'm fine with consistency, but maybe the thing to do here then is to
>> > ensure
>> > > that we definitely log the "effective" or "derived" config before
>> using
>> > it
>> > > so there is at least some useful trace of what we actually used that
>> can
>> > be
>> > > helpful in debugging.
>> >
>>
>
>

Re: [DISCUSS] KIP-249: Add Delegation Token Operations to Kafka Admin Client

[Manikumar]

Hi all,

We want to include this KIP in the upcoming 1.1.0 release.
Please let me know if there are any other comments.

If there are no more comments, I'd like to start vote on this KIP.

Thanks,

On Wed, Jan 17, 2018 at 8:20 AM, Manikumar 
wrote:

> Hi, Jun,
>
> Thanks for the review.
>
> 1.  Yes,  We can pass hmac  as byte[]. Updated the KIP
> 2.  Yes,  describeDelegationToken() returns all the user owned tokens and
> tokens where user have Describe permission.
>  Added a comment to KIP.
> 3.  updated the KIP with possible exceptions.
>
>
> Thanks,
>
>
> On Wed, Jan 17, 2018 at 6:45 AM, Jun Rao  wrote:
>
>> Hi, Mani,
>>
>> Thanks for the KIP. Looks good to me overhead. Just a couple of minor
>> comments below.
>>
>> 1. Should hmac be of type ByteBuffer? We return hmac as byte[] in
>> DelegationToken.
>> So, it seems it's more consistent to pass in hmac as byte[] too.
>> 2. Does describeDelegationToken() return all tokens?
>> 3. As Ted mentioned, it would be useful to include the exceptions that can
>> be thrown in the new apis.
>>
>> Jun
>>
>> On Tue, Jan 16, 2018 at 10:03 AM, Manikumar 
>> wrote:
>>
>> > Hi all,
>> >
>> > I have created a KIP to add delegation token operations to Java Admin
>> > Client.
>> > This KIP proposes new API additions to admin client. There are no new
>> wire
>> > protocol changes.
>> >
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > 249%3A+Add+Delegation+Token+Operations+to+KafkaAdminClient
>> >
>> >
>> > Feedback and suggestions are welcome.
>> >
>> > Thanks
>> > Manikumar
>> >
>>
>
>