Build failed in Jenkins: kafka-0.10.1-jdk7 #127

[Apache Jenkins Server]

See 


Changes:

[me] MINOR: Fix missing wait_until import from bad cherry-pick

--
[...truncated 585.60 KB...]
org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueStoreIteratorTest
 > shouldIterateOverRange PASSED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueStoreIteratorTest
 > shouldIgnoreIfDeletedInCacheButExistsInStore STARTED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueStoreIteratorTest
 > shouldIgnoreIfDeletedInCacheButExistsInStore PASSED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueStoreIteratorTest
 > shouldSkipAllDeletedFromCache STARTED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueStoreIteratorTest
 > shouldSkipAllDeletedFromCache PASSED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueStoreIteratorTest
 > shouldSkipLargerDeletedCacheValue STARTED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueStoreIteratorTest
 > shouldSkipLargerDeletedCacheValue PASSED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueStoreIteratorTest
 > shouldSkipSmallerDeletedCachedValue STARTED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueStoreIteratorTest
 > shouldSkipSmallerDeletedCachedValue PASSED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > 
shouldUseCustomRocksDbConfigSetter STARTED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > 
shouldUseCustomRocksDbConfigSetter PASSED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > 
shouldPerformAllQueriesWithCachingDisabled STARTED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > 
shouldPerformAllQueriesWithCachingDisabled PASSED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > 
shouldPerformRangeQueriesWithCachingDisabled STARTED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > 
shouldPerformRangeQueriesWithCachingDisabled PASSED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > 
shouldCloseOpenIteratorsWhenStoreClosedAndThrowInvalidStateStoreOnHasNextAndNext
 STARTED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > 
shouldCloseOpenIteratorsWhenStoreClosedAndThrowInvalidStateStoreOnHasNextAndNext
 PASSED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > testSize 
STARTED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > testSize 
PASSED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > 
testPutIfAbsent STARTED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > 
testPutIfAbsent PASSED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > 
testRestoreWithDefaultSerdes STARTED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > 
testRestoreWithDefaultSerdes PASSED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > testRestore 
STARTED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > testRestore 
PASSED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > 
testPutGetRange STARTED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > 
testPutGetRange PASSED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > 
testPutGetRangeWithDefaultSerdes STARTED

org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > 
testPutGetRangeWithDefaultSerdes PASSED

org.apache.kafka.streams.state.internals.DelegatingPeekingKeyValueIteratorTest 
> shouldPeekNext STARTED

org.apache.kafka.streams.state.internals.DelegatingPeekingKeyValueIteratorTest 
> shouldPeekNext PASSED

org.apache.kafka.streams.state.internals.DelegatingPeekingKeyValueIteratorTest 
> shouldPeekAndIterate STARTED

org.apache.kafka.streams.state.internals.DelegatingPeekingKeyValueIteratorTest 
> shouldPeekAndIterate PASSED

org.apache.kafka.streams.state.internals.DelegatingPeekingKeyValueIteratorTest 
> shouldThrowNoSuchElementWhenNoMoreItemsLeftAndPeekNextCalled STARTED

org.apache.kafka.streams.state.internals.DelegatingPeekingKeyValueIteratorTest 
> shouldThrowNoSuchElementWhenNoMoreItemsLeftAndPeekNextCalled PASSED

org.apache.kafka.streams.state.internals.DelegatingPeekingKeyValueIteratorTest 
> shouldThrowNoSuchElementWhenNoMoreItemsLeftAndNextCalled STARTED

org.apache.kafka.streams.state.internals.DelegatingPeekingKeyValueIteratorTest 
> shouldThrowNoSuchElementWhenNoMoreItemsLeftAndNextCalled PASSED

org.apache.kafka.streams.state.internals.OffsetCheckpointTest > testReadWrite 
STARTED

org.apache.kafka.streams.state.internals.OffsetCheckpointTest > testReadWrite 
PASSED

org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest > 
testCachingEnabled STARTED

org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest > 
testCachingEnabled PASSED

org.ap

[GitHub] kafka pull request #3638: MINOR: Fix missing wait_until import from bad cher...

[ewencp]

Github user ewencp closed the pull request at:

https://github.com/apache/kafka/pull/3638


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

[Apurva Mehta]

Hi Colin,

The KIP looks good to me. In your latest proposal, the change of state
would be captured as followed in the metrics for groups using Kafka for
membership management:

PreparingRebalance -> CompletingRebalance -> Stable -> Dead?

If a group is just being used to store offsets, then it is always Empty?

If so, this makes sense to me.

Thanks,
Apurva

On Mon, Aug 7, 2017 at 5:09 PM, Colin McCabe  wrote:

> How about PreparingRebalance / CompletingRebalance?
>
> cheers,
> Colin
>
>
> On Fri, Aug 4, 2017, at 09:03, Ismael Juma wrote:
> > I agree that we should make them consistent. I think RebalanceJoin and
> > RebalanceAssignment are reasonable names. I think they are a bit more
> > descriptive than `PreparingRebalance` and `CompletingRebalance`. If we
> > need
> > to add more states, it seems a little easier to do if the states are a
> > bit
> > more descriptive. I am OK with either of the 2 options as I think they
> > are
> > both better than the status quo.
> >
> > Ismael
> >
> > On Fri, Aug 4, 2017 at 4:52 PM, Jason Gustafson 
> > wrote:
> >
> > > Hey Guozhang,
> > >
> > > Usually I think such naming inconsistencies are best avoided. It adds
> > > another level of confusion for people who have to dip into the code,
> figure
> > > out a problem, and ultimately explain it. Since we already have the
> > > PreparingRebalance state, maybe we could just rename the AwaitingSync
> state
> > > to CompletingRebalance?
> > >
> > > -Jason
> > >
> > > On Thu, Aug 3, 2017 at 6:09 PM, Guozhang Wang 
> wrote:
> > >
> > > > From an ops person's view who are mostly likely watching the metrics
> > > these
> > > > names may not be very clear as people may not know the internals
> well.
> > > I'd
> > > > prefer PrepareRebalance and CompleteRebalance since they may be
> easier to
> > > > understand thought not 100 percent accurately match to internal
> > > > implementation.
> > > >
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Thu, Aug 3, 2017 at 4:14 PM, Jason Gustafson 
> > > > wrote:
> > > >
> > > > > Hey Colin, Guozhang,
> > > > >
> > > > > I agree the current state names are not ideal for end users. I
> tend to
> > > > see
> > > > > the rebalance as joining the group and receiving the assignment.
> Maybe
> > > > the
> > > > > states could be named in those terms? For example: RebalanceJoin
> and
> > > > > RebalanceAssignment. What do you think?
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Fri, Jul 28, 2017 at 11:18 AM, Guozhang Wang <
> wangg...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I feel we can change `AwaitSync` to `completeRebalance` while
> keeping
> > > > the
> > > > > > other as is.
> > > > > >
> > > > > > cc Jason?
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > > On Fri, Jul 28, 2017 at 10:08 AM, Colin McCabe <
> cmcc...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > >> Thanks for the explanation.  I guess maybe we should just keep
> the
> > > > group
> > > > > >> names as they are, then?
> > > > > >>
> > > > > >> best,
> > > > > >> Colin
> > > > > >>
> > > > > >>
> > > > > >> On Wed, Jul 26, 2017, at 11:25, Guozhang Wang wrote:
> > > > > >> > To me `PreparingRebalance` sounds better than
> `StartingRebalance`
> > > > > since
> > > > > >> > only by the end of that stage we have formed a new group. More
> > > > > >> > specifically, this this the workflow from the coordinator's
> point
> > > of
> > > > > >> > view:
> > > > > >> >
> > > > > >> > 1. decided to trigger a rebalance, enter PreparingRebalance
> phase;
> > > > > >> >   |
> > > > > >> >   |   send out error code for all heartbeat
> > > reponses
> > > > > >> >  \|/
> > > > > >> >   |
> > > > > >> >   |   waiting for join group requests from
> members
> > > > > >> >  \|/
> > > > > >> > 2. formed a new group, increment the generation number, now
> start
> > > > > >> > rebalancing, entering AwaitSync phase:
> > > > > >> >   |
> > > > > >> >   |   send out the join group responses for
> > > whoever
> > > > > >> > requested join
> > > > > >> >  \|/
> > > > > >> >   |
> > > > > >> >   |   waiting for the sync group request from
> the
> > > > > leader
> > > > > >> >  \|/
> > > > > >> > 3. received assignment from the leader; the rebalance has
> ended,
> > > > start
> > > > > >> > ticking for all members, entering Stable phase.
> > > > > >> >   |
> > > > > >> >   |   for whoever else sending the sync group
> > > > request,
> > > > > >> > reply with the assignment
> > > > > >> >  \|/
> > > > > >> >
> > > > > >> > So from the coordinator's point of view the rebalance starts
> at
> > > > > >> beginning
> > > > > >> > of step 2 and ends at beginning of step 3. Maybe we can rename
> > > > > >> > `AwaitSync`
> > > > > >> > itself to `CompletingRebalance`.
> > > > > >> >
> > > > > >> > Guozhan

Re: [DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

[Colin McCabe]

How about PreparingRebalance / CompletingRebalance?

cheers,
Colin


On Fri, Aug 4, 2017, at 09:03, Ismael Juma wrote:
> I agree that we should make them consistent. I think RebalanceJoin and
> RebalanceAssignment are reasonable names. I think they are a bit more
> descriptive than `PreparingRebalance` and `CompletingRebalance`. If we
> need
> to add more states, it seems a little easier to do if the states are a
> bit
> more descriptive. I am OK with either of the 2 options as I think they
> are
> both better than the status quo.
> 
> Ismael
> 
> On Fri, Aug 4, 2017 at 4:52 PM, Jason Gustafson 
> wrote:
> 
> > Hey Guozhang,
> >
> > Usually I think such naming inconsistencies are best avoided. It adds
> > another level of confusion for people who have to dip into the code, figure
> > out a problem, and ultimately explain it. Since we already have the
> > PreparingRebalance state, maybe we could just rename the AwaitingSync state
> > to CompletingRebalance?
> >
> > -Jason
> >
> > On Thu, Aug 3, 2017 at 6:09 PM, Guozhang Wang  wrote:
> >
> > > From an ops person's view who are mostly likely watching the metrics
> > these
> > > names may not be very clear as people may not know the internals well.
> > I'd
> > > prefer PrepareRebalance and CompleteRebalance since they may be easier to
> > > understand thought not 100 percent accurately match to internal
> > > implementation.
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Thu, Aug 3, 2017 at 4:14 PM, Jason Gustafson 
> > > wrote:
> > >
> > > > Hey Colin, Guozhang,
> > > >
> > > > I agree the current state names are not ideal for end users. I tend to
> > > see
> > > > the rebalance as joining the group and receiving the assignment. Maybe
> > > the
> > > > states could be named in those terms? For example: RebalanceJoin and
> > > > RebalanceAssignment. What do you think?
> > > >
> > > > -Jason
> > > >
> > > > On Fri, Jul 28, 2017 at 11:18 AM, Guozhang Wang 
> > > > wrote:
> > > >
> > > > > I feel we can change `AwaitSync` to `completeRebalance` while keeping
> > > the
> > > > > other as is.
> > > > >
> > > > > cc Jason?
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Fri, Jul 28, 2017 at 10:08 AM, Colin McCabe 
> > > > wrote:
> > > > >
> > > > >> Thanks for the explanation.  I guess maybe we should just keep the
> > > group
> > > > >> names as they are, then?
> > > > >>
> > > > >> best,
> > > > >> Colin
> > > > >>
> > > > >>
> > > > >> On Wed, Jul 26, 2017, at 11:25, Guozhang Wang wrote:
> > > > >> > To me `PreparingRebalance` sounds better than `StartingRebalance`
> > > > since
> > > > >> > only by the end of that stage we have formed a new group. More
> > > > >> > specifically, this this the workflow from the coordinator's point
> > of
> > > > >> > view:
> > > > >> >
> > > > >> > 1. decided to trigger a rebalance, enter PreparingRebalance phase;
> > > > >> >   |
> > > > >> >   |   send out error code for all heartbeat
> > reponses
> > > > >> >  \|/
> > > > >> >   |
> > > > >> >   |   waiting for join group requests from members
> > > > >> >  \|/
> > > > >> > 2. formed a new group, increment the generation number, now start
> > > > >> > rebalancing, entering AwaitSync phase:
> > > > >> >   |
> > > > >> >   |   send out the join group responses for
> > whoever
> > > > >> > requested join
> > > > >> >  \|/
> > > > >> >   |
> > > > >> >   |   waiting for the sync group request from the
> > > > leader
> > > > >> >  \|/
> > > > >> > 3. received assignment from the leader; the rebalance has ended,
> > > start
> > > > >> > ticking for all members, entering Stable phase.
> > > > >> >   |
> > > > >> >   |   for whoever else sending the sync group
> > > request,
> > > > >> > reply with the assignment
> > > > >> >  \|/
> > > > >> >
> > > > >> > So from the coordinator's point of view the rebalance starts at
> > > > >> beginning
> > > > >> > of step 2 and ends at beginning of step 3. Maybe we can rename
> > > > >> > `AwaitSync`
> > > > >> > itself to `CompletingRebalance`.
> > > > >> >
> > > > >> > Guozhang
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Tue, Jul 25, 2017 at 6:44 AM, Ismael Juma 
> > > > wrote:
> > > > >> >
> > > > >> > > Hi Guozhang,
> > > > >> > >
> > > > >> > > Thanks for the clarification. The naming does seem a bit
> > unclear.
> > > > >> Maybe
> > > > >> > > `PreparingRebalance` could be `StartingRebalance` or something
> > > that
> > > > >> makes
> > > > >> > > it clear that it is part of the rebalance instead of a step
> > before
> > > > the
> > > > >> > > actual rebalance. `AwaitingSync` could also be
> > > > `CompletingRebalance`,
> > > > >> but
> > > > >> > > not sure if that's better.
> > > > >> > >
> > > > >> > > Ismael
> > > > >> > >
> > > > >> > > On Mon, Jul 24, 2017 at 7:02 PM, Guozhang Wang <

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

[Apurva Mehta]

Responses inline:

On Mon, Aug 7, 2017 at 9:37 AM, Sumant Tambe  wrote:

>
> >
> > However, one thing which has not come out of the JIRA discussion is the
> > actual use cases for batch expiry.
>
> There are two usecases I can think of for batch expiry mechanism
> irrespective of how we try to bound the time (batch.expiry.ms or
> max.message.delivery.wait.ms). Let's call it X.
>
> 1. A real-time app (e.g., periodic healthcheck producer, temperature sensor
> producer) has a soft upper bound on both message delivery and failure
> notification of message delivery. In both cases, it wants to know. Such an
> app does not close the producer on the first error reported (due to batch
> expiry) because there's data lined up right behind. It's ok to lose a few
> samples of temperature measurement (IoT scenario). So it simply drops it
> and moves on. May be when drop rate is like 70% it would close it. Such an
> app may use acks=0. In this case, X will have some value in single digit
> minutes. But X=MAX_LONG is not suitable.
>

I guess my question is: batches would only start expiring if partition(s)
are unavailable. What would 'moving on' mean for a real time app in this
case? How would that help? After expiring one set of batches, the next set
would also be expired until the cluster comes back. So what is achieved by
expiring the batches at all?


>
> 2. Today we run KMM in Linkedin as if batch.expiry==MAX_LONG. We expire
> under the condition: (!muted.contains(tp) && (isMetadataStale ||
> cluster.leaderFor(tp) == null)) In essence, as long as the partition is
> making progress (even if it's a trickle), the producer keeps on going.
> We've other internal systems to detect whether a pipeline is making
> *sufficient* progress or not. We're not dependent on the producer to tell
> us that it's not making progress on a certain partition.
>
> This is less than ideal though. We would be happy to configure
> batch.expiry.ms to 1800,000 or so and upon notification of expiry restart
> the process and what not. It can tell also tell us which specific
> partitions of a specific topic is falling behind. We achieve a similar
> effect via alternative mechanisms.


I presume you are referring to partitions here? If a some partitions are
unavailable, what could mirror maker do with that knowledge? Presumably
focus on the partitions which are online. But does that really help?  Today
the expiry message already contains the partition information, is that
being used? If not, how would accurate expiry times with something like
batch.expiry.ms change that fact?


>
> > Also, the KIP document states the
> > following:
> >
> > *The per message timeout is easy to compute - linger.ms
> > >  + (retries + 1) * request.timeout.ms
> > > ". *This is false.
> >
>
> > Why is the statement false? Doesn't that provide an accurate upperbound
> on
> > the timeout for a produce request today?
> >
> The KIP-91 write-up describes the reasons why. Just reiterating the reason:
> "the condition that if the metadata for a partition is known then we do not
> expire its batches even if they are ready".  Do you not agree with the
> explanation? If not, what part?
>
>
Unless I have missed something, the producer doesn't seem to rely on the
metadata being known at least since https://github.com/apache/kafka/pull/503.
The current code to decide whether a batch should be expired is here :
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/ProducerBatch.java#L298
.

There seems to be no relationship with cluster metadata availability or
staleness. Expiry is just based on the time since the batch has been ready.
Please correct me if I am wrong.



> >
> > In this spirit, it might make sense to clarify the use case that
> motivates
> > this additional setting. For instance, with this new configuration, how
> > would your existing application handle a batch expired exception?
>
> Again, a real-time app would just move on. KMM would halt. Any
> order-sensitive app which needs to provide durability guarantees would
> halt.
>

I had a question about the real time app earlier. But KMM and other
applications can already halt today once they get backed up without this
new proposed config. How does being slightly more precise of *when* to halt
change anything about these semantics in a meaningful way?

Thanks,
Apurva

[GitHub] kafka pull request #3640: KAFKA-5701: fix flaky unit test

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3640


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

[Dong Lin]

Hey Tom,

Yeah I agree with you that the total disk capacity can be useful
particularly if it is different across brokers but it is probably of
limited use in most cases. I also expect that most users would have their
own customized tool across to determine the new partition reassignment
after retrieving the partition distribution using DescribeDirsRequest. And
that customized tool can probably be easily provided with the configuration
(e.g. disk capacity, IO parameters) of the disks in the cluster when user
runs it.

I am relatively neural on whether or not we should add this field. If there
is no strong reason to add this field, I will add it if one or more
committer recommends to do this.

Thanks,
Dong



On Mon, Aug 7, 2017 at 1:01 PM, Tom Bentley  wrote:

> Hi Dong,
>
> The reason I thought this would be useful is it seems likely to me that
> people will want to write tools to help them generate allocations. If, as
> you say, all the brokers and all the disks are the same size, then it's not
> too difficult to tell the tool the size of the disk. But if they're not the
> same, then using the tool becomes a lot harder. Obviously if the size of
> the disk is included in the DescribeDirsResponse then you can literally
> just point the tool at the cluster.
>
> On the other hand, it seems likely that tools might also want to take into
> account other things when trying to find a good assignment (per-device IO
> for example) between the disks on a broker, so maybe including the total
> disk capacity is only of limited use.
>
> Cheers,
>
> Tom
>
> On 7 August 2017 at 17:54, Dong Lin  wrote:
>
> > Hey Tom,
> >
> > Good question. We have actually considered having DescribeDirsResponse
> > provide the capacity of each disk as well. This was not included because
> we
> > believe Kafka cluster admin will always configure all brokers with same
> > number of disks of the same size. This is because it is generally easier
> to
> > manager a homogeneous cluster. If this is not the case then I think we
> > should include this information in the response.
> >
> > Thanks,
> > Dong
> >
> >
> > On Mon, Aug 7, 2017 at 3:44 AM, Tom Bentley 
> wrote:
> >
> > > Hi Dong,
> > >
> > > Your comments on KIP-179 prompted me to look at KIP-113, and I have a
> > > question:
> > >
> > > AFAICS the DescribeDirsResponse (via ReplicaInfo) can be used to get
> the
> > > size of a partition on a disk, but I don't see a mechanism for knowing
> > the
> > > total capacity of a disk (and/or the free capacity of a disk). That
> would
> > > be very useful information to have to help figure out that certain
> > > assignments are impossible, for instance. Is there a reason you've left
> > > this out?
> > >
> > > Cheers,
> > >
> > > Tom
> > >
> > > On 4 August 2017 at 18:47, Dong Lin  wrote:
> > >
> > > > Hey Ismael,
> > > >
> > > > Thanks for the comments! Here are my answers:
> > > >
> > > > 1. Yes it has been considered. Here are the reasons why we don't do
> it
> > > > through controller.
> > > >
> > > > - There can be use-cases where we only want to rebalance the load of
> > log
> > > > directories on a given broker. It seems unnecessary to go through
> > > > controller in this case.
> > > >
> > > >  - If controller is responsible for sending ChangeReplicaDirRequest,
> > and
> > > if
> > > > the user-specified log directory is either invalid or offline, then
> > > > controller probably needs a way to tell user that the partition
> > > > reassignment has failed. We currently don't have a way to do this
> since
> > > > kafka-reassign-partition.sh simply creates the reassignment znode
> > without
> > > > waiting for response. I am not sure that is a good solution to this.
> > > >
> > > > - If controller is responsible for sending ChangeReplicaDirRequest,
> the
> > > > controller logic would be more complicated because controller needs
> to
> > > > first send ChangeReplicaRequest so that the broker memorize the
> > partition
> > > > -> log directory mapping, send LeaderAndIsrRequest, and keep sending
> > > > ChangeReplicaDirRequest (just in case broker restarted) until replica
> > is
> > > > created. Note that the last step needs repeat and timeout as the
> > proposed
> > > > in the KIP-113.
> > > >
> > > > Overall I think this adds quite a bit complexity to controller and we
> > > > probably want to do this only if there is strong clear of doing so.
> > > > Currently in KIP-113 the kafka-reassign-partitions.sh is responsible
> > for
> > > > sending ChangeReplicaDirRequest with repeat and provides error to
> user
> > if
> > > > it either fails or timeout. It seems to be much simpler and user
> > > shouldn't
> > > > care whether it is done through controller.
> > > >
> > > > And thanks for the suggestion. I will add this to the Rejected
> > > Alternative
> > > > Section in the KIP-113.
> > > >
> > > > 2) I think user needs to be able to specify different log directories
> > for
> > > > the replicas of the same partition in order to rebalance load acr

[jira] [Resolved] (KAFKA-5681) jarAll does not build all scala versions anymore.

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin resolved KAFKA-5681.
-
Resolution: Won't Fix

Discussed in the PR. Closing the ticket as won't fix.

> jarAll does not build all scala versions anymore.
> -
>
> Key: KAFKA-5681
> URL: https://issues.apache.org/jira/browse/KAFKA-5681
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.11.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.11.0.1
>
>
> ./gradlew jarAll no longer builds jars for all scala versions. We should use 
> {{availableScalaVersions}} instead of {{defaultScalaVersions}} when build. We 
> probably should consider backporting the fix to 0.11.0.0.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3628: KAFKA-5681; jarAll should build all scala versions...

[becketqin]

Github user becketqin closed the pull request at:

https://github.com/apache/kafka/pull/3628


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Kafka Client for Swift 4

[Kellan Burket Cummings]

Thanks, Jason,
It's kellan.burket
- Kellan

On Thu, Aug 3, 2017 at 6:39 PM, Jason Gustafson  wrote:

> Hi Kellan,
>
> Looks cool. If you tell me your user id, I can give you access to edit the
> wiki directly.
>
> -Jason
>
> On Fri, Jul 28, 2017 at 1:41 PM, Kellan Burket Cummings <
> kellan.bur...@gmail.com> wrote:
>
> > Here it is: https://github.com/kellanburket/franz
> >
> > Can someone with access to the Wiki please add it to the third-party
> > clients page?
> >
> > Thanks,
> > Kellan
> >
>

Re: [DISCUSS] KIP-184 Rename LogCleaner and related classes to LogCompactor

[Guozhang Wang]

Thanks for the KIP proposal,

I thought one suggestion before this discussion is to deprecate the "
log.cleaner.enable" and always turn on compaction for those topics that
have compact policies?


Guozhang

On Sat, Aug 5, 2017 at 9:36 AM, Pranav Maniar  wrote:

> Hi All,
>
> Following a discussion on JIRA KAFKA-1944
>  . I have created
> KIP-184
>  184%3A+Rename+LogCleaner+and+related+classes+to+LogCompactor>
> as
> it will require configuration change.
>
> As per the process I am starting Discussion on mail thread for KIP-184.
>
> Renaming of configuration "log.cleaner.enable" is discussed on KAFKA-1944.
> But other log.cleaner configuration also seems to be used by cleaner only.
> So to maintain naming consistency, I have proposed to rename all these
> configuration.
>
> Please provide your suggestion/views for the same. Thanks !
>
>
> Thanks,
> Pranav
>



-- 
-- Guozhang

[jira] [Created] (KAFKA-5709) Improve logging to include errors from state-change log in server log

[Alla Tumarkin (JIRA)]

Alla Tumarkin created KAFKA-5709:


 Summary: Improve logging to include errors from state-change log 
in server log
 Key: KAFKA-5709
 URL: https://issues.apache.org/jira/browse/KAFKA-5709
 Project: Kafka
  Issue Type: Improvement
  Components: log
Affects Versions: 0.10.2.0
Reporter: Alla Tumarkin


Problem
The following message was generated over and over again when running 
kafka-console-producer or kafka-console-consumer with SSL and ACLs enabled
{code}
WARN Error while fetching metadata with correlation id 1 : 
{test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)` endlessly 
when I run kafka-console-producer or kafka-console-consumer
{code}
however server log (or authorizer log) did not indicate any problem.

Background
1) Initially, security.inter.broker.protocol setting was missing in 
server.properties, so connection was falling back to using plaintext port 
(9092), and only state-change.log actually contained the underlying error:
{code}
[2017-08-04 13:40:48,536] TRACE Controller 0 epoch 6 received response 
{error_code=31,partitions=[{topic=test,partition=0,error_code=31},{topic=__confluent.support.metrics,partition=0,error_code=31}]}
 for a request sent to broker localhost:9092 (id: 0 rack: null) 
(state.change.logger)
{code}
as per
https://kafka.apache.org/protocol#protocol_error_codes
{code}
CLUSTER_AUTHORIZATION_FAILED31  False   Cluster authorization failed.
{code}

2) After setting "security.inter.broker.protocol=SSL" the port changed to 
secure (9093) yet the error in state-change log did not go away:
{code}
[2017-08-04 13:49:38,462] TRACE Controller 0 epoch 7 received response 
{error_code=31} for a request sent to broker localhost:9093 (id: 0 rack: null) 
(state.change.logger)
{code}
and LEADER_NOT_AVAILABLE was still generated.

This time though, kafka-authorizer.log had a better indication of the problem:
{code}
[2017-08-04 18:17:46,770] DEBUG No acl found for resource 
Cluster:kafka-cluster, authorized = false (kafka.authorizer.logger)
[2017-08-04 18:17:46,770] DEBUG Principal = 
User:CN=Unknown,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown is Denied 
Operation = ClusterAction from host = 127.0.0.1 on resource = 
Cluster:kafka-cluster (kafka.authorizer.logger)
{code}
The issue being that topic metadata is not propagated successfully from 
controller to broker since the broker user doesn't have ClusterAction 
permission.
Fixed by
{code}
bin/kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 --add 
--allow-principal 
"User:CN=Unknown,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown" 
--operation ClusterAction --cluster
{code}

Request
The debugging is tricky since the controller to broker logging is done in 
controller/state-change log, not in the main server log.
We need to improve the logging on this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

[Tom Bentley]

Hi Dong,

The reason I thought this would be useful is it seems likely to me that
people will want to write tools to help them generate allocations. If, as
you say, all the brokers and all the disks are the same size, then it's not
too difficult to tell the tool the size of the disk. But if they're not the
same, then using the tool becomes a lot harder. Obviously if the size of
the disk is included in the DescribeDirsResponse then you can literally
just point the tool at the cluster.

On the other hand, it seems likely that tools might also want to take into
account other things when trying to find a good assignment (per-device IO
for example) between the disks on a broker, so maybe including the total
disk capacity is only of limited use.

Cheers,

Tom

On 7 August 2017 at 17:54, Dong Lin  wrote:

> Hey Tom,
>
> Good question. We have actually considered having DescribeDirsResponse
> provide the capacity of each disk as well. This was not included because we
> believe Kafka cluster admin will always configure all brokers with same
> number of disks of the same size. This is because it is generally easier to
> manager a homogeneous cluster. If this is not the case then I think we
> should include this information in the response.
>
> Thanks,
> Dong
>
>
> On Mon, Aug 7, 2017 at 3:44 AM, Tom Bentley  wrote:
>
> > Hi Dong,
> >
> > Your comments on KIP-179 prompted me to look at KIP-113, and I have a
> > question:
> >
> > AFAICS the DescribeDirsResponse (via ReplicaInfo) can be used to get the
> > size of a partition on a disk, but I don't see a mechanism for knowing
> the
> > total capacity of a disk (and/or the free capacity of a disk). That would
> > be very useful information to have to help figure out that certain
> > assignments are impossible, for instance. Is there a reason you've left
> > this out?
> >
> > Cheers,
> >
> > Tom
> >
> > On 4 August 2017 at 18:47, Dong Lin  wrote:
> >
> > > Hey Ismael,
> > >
> > > Thanks for the comments! Here are my answers:
> > >
> > > 1. Yes it has been considered. Here are the reasons why we don't do it
> > > through controller.
> > >
> > > - There can be use-cases where we only want to rebalance the load of
> log
> > > directories on a given broker. It seems unnecessary to go through
> > > controller in this case.
> > >
> > >  - If controller is responsible for sending ChangeReplicaDirRequest,
> and
> > if
> > > the user-specified log directory is either invalid or offline, then
> > > controller probably needs a way to tell user that the partition
> > > reassignment has failed. We currently don't have a way to do this since
> > > kafka-reassign-partition.sh simply creates the reassignment znode
> without
> > > waiting for response. I am not sure that is a good solution to this.
> > >
> > > - If controller is responsible for sending ChangeReplicaDirRequest, the
> > > controller logic would be more complicated because controller needs to
> > > first send ChangeReplicaRequest so that the broker memorize the
> partition
> > > -> log directory mapping, send LeaderAndIsrRequest, and keep sending
> > > ChangeReplicaDirRequest (just in case broker restarted) until replica
> is
> > > created. Note that the last step needs repeat and timeout as the
> proposed
> > > in the KIP-113.
> > >
> > > Overall I think this adds quite a bit complexity to controller and we
> > > probably want to do this only if there is strong clear of doing so.
> > > Currently in KIP-113 the kafka-reassign-partitions.sh is responsible
> for
> > > sending ChangeReplicaDirRequest with repeat and provides error to user
> if
> > > it either fails or timeout. It seems to be much simpler and user
> > shouldn't
> > > care whether it is done through controller.
> > >
> > > And thanks for the suggestion. I will add this to the Rejected
> > Alternative
> > > Section in the KIP-113.
> > >
> > > 2) I think user needs to be able to specify different log directories
> for
> > > the replicas of the same partition in order to rebalance load across
> log
> > > directories of all brokers. I am not sure I understand the question.
> Can
> > > you explain a bit more why "that the log directory has to be the same
> for
> > > all replicas of a given partition"?
> > >
> > > 3) Good point. I think the alterReplicaDir is a better than
> > > changeReplicaDir for the reason you provided. I will also update names
> of
> > > the request/response as well in the KIP.
> > >
> > >
> > > Thanks,
> > > Dong
> > >
> > > On Fri, Aug 4, 2017 at 9:49 AM, Ismael Juma  wrote:
> > >
> > > > Thanks Dong. I have a few initial questions, sorry if I it has been
> > > > discussed and I missed it.
> > > >
> > > > 1. The KIP suggests that the reassignment tool is responsible for
> > sending
> > > > the ChangeReplicaDirRequests to the relevant brokers. I had imagined
> > that
> > > > this would be done by the Controller, like the rest of the
> reassignment
> > > > process. Was this considered? If so, it would be good to include the
> > > > details of why

[GitHub] kafka pull request #3641: KAFKA-5704 Corrected Connect distributed startup b...

[rhauch]

GitHub user rhauch opened a pull request:

https://github.com/apache/kafka/pull/3641

KAFKA-5704 Corrected Connect distributed startup behavior to allow older 
brokers to auto-create topics

When a Connect distributed worker starts up talking with broker versions 
0.10.1.0 and later, it will use the AdminClient to look for the internal topics 
and attempt to create them if they are missing. Although the AdminClient was 
added in 0.11.0.0, the AdminClient uses APIs to create topics that existed in 
0.10.1.0 and later. This feature works as expected when Connect uses a broker 
version 0.10.1.0 or later.

However, when a Connect distributed worker starts up using a broker older 
than 0.10.1.0, the AdminClient is not able to find the required APIs and thus 
will throw an UnsupportedVersionException. Unfortunately, this exception is not 
caught and instead causes the Connect worker to fail even when the topics 
already exist.

This change handles the UnsupportedVersionException by logging a debug 
message and doing nothing. The existing producer logic will get information 
about the topics, which will cause the broker to create them if they don’t 
exist and broker auto-creation of topics is enabled. This is the same behavior 
that existed prior to 0.11.0.0, and so this change restores that behavior for 
brokers older than 0.10.1.0.

This change also adds a system test that verifies Connect works with a 
variety of brokers and is able to run source and sink connectors. The test 
verifies that Connect can read from the internal topics when the connectors are 
restarted.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rhauch/kafka kafka-5704

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3641.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3641


commit 0d45fd113eeaff3844742181191db2cc508353fd
Author: Randall Hauch 
Date:   2017-08-07T19:32:29Z

KAFKA-5704 Corrected Connect distributed startup behavior to allow older 
brokers to auto-create topics

When a Connect distributed worker starts up talking with broker versions 
0.10.1.0 and later, it will use the AdminClient to look for the internal topics 
and attempt to create them if they are missing. Although the AdminClient was 
added in 0.11.0.0, the AdminClient uses APIs to create topics that existed in 
0.10.1.0 and later. This feature works as expected when Connect uses a broker 
version 0.10.1.0 or later.

However, when a Connect distributed worker starts up using a broker older 
than 0.10.1.0, the AdminClient is not able to find the required APIs and thus 
will throw an UnsupportedVersionException. Unfortunately, this exception is not 
caught and instead causes the Connect worker to fail even when the topics 
already exist.

This change handles the UnsupportedVersionException by logging a debug 
message and doing nothing. The existing producer logic will get information 
about the topics, which will cause the broker to create them if they don’t 
exist and broker auto-creation of topics is enabled. This is the same behavior 
that existed prior to 0.11.0.0, and so this change restores that behavior for 
brokers older than 0.10.1.0.

This change also adds a system test that verifies Connect works with a 
variety of brokers and is able to run source and sink connectors. The test 
verifies that Connect can read from the internal topics when the connectors are 
restarted.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3640: Kafka 5701 fix flaky unit test

[bbejeck]

GitHub user bbejeck opened a pull request:

https://github.com/apache/kafka/pull/3640

Kafka 5701 fix flaky unit test

1. Remove separate thread from test failing periodically due to race 
condition.
2. Remove anonymous `AbstractNotifyingBatchingRestoreCallback` and set as 
package private variable.  Class is static so it has to initialize it's 
dependency on `RocksDBStore`

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bbejeck/kafka KAFKA-5701_fix_flaky_unit_test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3640.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3640






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-5705) Kafka Server start failed and reports "unsafe memory access operation"

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-5705.

Resolution: Duplicate

Marking this as a duplicate of KAFKA-5628 since the exception trace looks 
identical. Please reopen if you disagree.

> Kafka Server start failed and reports "unsafe memory access operation"
> --
>
> Key: KAFKA-5705
> URL: https://issues.apache.org/jira/browse/KAFKA-5705
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.10.2.0
>Reporter: Chen He
>
> [2017-08-02 15:50:23,361] FATAL Fatal error during KafkaServerStartable 
> startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
> java.lang.InternalError: a fault occurred in a recent unsafe memory access 
> operation in compiled Java code
> at 
> kafka.log.TimeIndex$$anonfun$maybeAppend$1.apply$mcV$sp(TimeIndex.scala:128)
> at 
> kafka.log.TimeIndex$$anonfun$maybeAppend$1.apply(TimeIndex.scala:107)
> at 
> kafka.log.TimeIndex$$anonfun$maybeAppend$1.apply(TimeIndex.scala:107)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:213)
> at kafka.log.TimeIndex.maybeAppend(TimeIndex.scala:107)
> at kafka.log.LogSegment.recover(LogSegment.scala:252)
> at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:231)
> at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:188)
> at 
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
> at 
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
> at 
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
> at kafka.log.Log.loadSegments(Log.scala:188)
> at kafka.log.Log.(Log.scala:116)
> at 
> kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:157)
> at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:57)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

[Dong Lin]

Hey Ismael,

Thanks much for your comments. Please see my reply inline.

On Mon, Aug 7, 2017 at 5:28 AM, Ismael Juma  wrote:

> Hi Dong,
>
> Thanks for the explanation. Comments inline.
>
> On Fri, Aug 4, 2017 at 6:47 PM, Dong Lin  wrote:
>
> > 1. Yes it has been considered. Here are the reasons why we don't do it
> > through controller.
> >
> > - There can be use-cases where we only want to rebalance the load of log
> > directories on a given broker. It seems unnecessary to go through
> > controller in this case.
> >
>
> Even though this is true, not sure how common it will be.
>

I think the frequency of the need to balance across disks on the same
broker will be considerably higher (e.g. 2X or more) than the frequency
needed to balance across brokers. This is because the underlying replica
has the same size distribution but the capacity of broker can be 10X as
much as the capacity of disk.

I don't think this is a strong argument for having this logic only in the
tool instead of controller. It is a nice to have feature if there is no
strong reason to do it in controller.


>
>  - If controller is responsible for sending ChangeReplicaDirRequest, and if
> > the user-specified log directory is either invalid or offline, then
> > controller probably needs a way to tell user that the partition
> > reassignment has failed. We currently don't have a way to do this since
> > kafka-reassign-partition.sh simply creates the reassignment znode without
> > waiting for response. I am not sure that is a good solution to this.
> >
>
> Since the JSON is provided by the user, we would ideally validate its
> contents before storing it. Why are the log directories different than the
> other information in the JSON?


I think there are two difference between the log directory field and other
fields in the JSON:

- The log directory field take much more bytes than other fields in the
reassignment znode. Due to the 1MB size limit of znode, Kafka admin
currently have to split a large reassignment into multiple smaller
reassignment which limits the number of partitions that can be moved
concurrently. Currently the reassignment znode has 1 integer for each
replica. The log directory will introduce probably 10+ characters for each
replica. This can significantly lower the number of partitions that can be
reassigned at the same time.

- All other fields in the reassignment znode can be found and verified by
the other znodes in the zookeeper. Thus controller only needs to register a
ZK listener to be notified once the reassignment completes. However, the
log directory of each replica is not in the zookeeper. The controller would
have to periodically sending DescribeDirsRequet to check whether the
replica has been successfully moved to the destination log directory.
Currently there is nothing like this in the controller logic. Ideally we
want to avoid adding this complexity and performance overhead in controller.



> - If controller is responsible for sending ChangeReplicaDirRequest, the
> > controller logic would be more complicated because controller needs to
> > first send ChangeReplicaRequest so that the broker memorize the partition
> > -> log directory mapping, send LeaderAndIsrRequest, and keep sending
> > ChangeReplicaDirRequest (just in case broker restarted) until replica is
> > created. Note that the last step needs repeat and timeout as the proposed
> > in the KIP-113.
> >
> > Overall I think this adds quite a bit complexity to controller and we
> > probably want to do this only if there is strong clear of doing so.
> > Currently in KIP-113 the kafka-reassign-partitions.sh is responsible for
> > sending ChangeReplicaDirRequest with repeat and provides error to user if
> > it either fails or timeout. It seems to be much simpler and user
> shouldn't
> > care whether it is done through controller.
> >
>
> If I understand correctly, the logic is the same in both cases, it's just a
> question of where it lives. The advantage of placing it in the Controller
> is that the whole reassignment logic is in one place (instead of split
> between the tool and the Controller). That seems easier to reason about.
>

It seems that the main motivation for putting this logic in controller is
to simplify the work for Kafka developer. I agree it is desirable to put
the logic in the same place. On the other hand we developer also want to
keep controller simple and efficient.

I actually did this in the original design but later I was convinced by Jun
that it is simpler to put the new logic in the reassignment tool. I think
we can put this logic in controller if we can find good solution to the
problems described above.


>
> Also, how do you think things would work in the context of KIP-179? Would
> the tool still invoke these requests or would it be done by the broker
> receiving the alterTopics/reassignPartitions protocol call?
>

My gut feel is that the tool will still invoke these requests. But I have a
few questions to KIP-179 before I c

[GitHub] kafka pull request #3639: MINOR: Standardize logging of Worker-level message...

[ewencp]

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/3639

MINOR: Standardize logging of Worker-level messages from Tasks and 
Connectors

This ensures all logs have the connector/task ID, whether tasks are source 
or sink, and formats them consistently.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
standardize-connector-task-logging

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3639.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3639


commit 9e16f2138c7104af5a7404d41de97d62a328d577
Author: Ewen Cheslack-Postava 
Date:   2017-08-07T17:36:06Z

MINOR: Standardize logging of Worker-level messages from Tasks and 
Connectors

This ensures all logs have the connector/task ID, whether tasks are source 
or sink, and formats them consistently.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3625: HOTFIX: fix for standby tasks using batching resto...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3625


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

[Dong Lin]

Hey Tom,

Good question. We have actually considered having DescribeDirsResponse
provide the capacity of each disk as well. This was not included because we
believe Kafka cluster admin will always configure all brokers with same
number of disks of the same size. This is because it is generally easier to
manager a homogeneous cluster. If this is not the case then I think we
should include this information in the response.

Thanks,
Dong


On Mon, Aug 7, 2017 at 3:44 AM, Tom Bentley  wrote:

> Hi Dong,
>
> Your comments on KIP-179 prompted me to look at KIP-113, and I have a
> question:
>
> AFAICS the DescribeDirsResponse (via ReplicaInfo) can be used to get the
> size of a partition on a disk, but I don't see a mechanism for knowing the
> total capacity of a disk (and/or the free capacity of a disk). That would
> be very useful information to have to help figure out that certain
> assignments are impossible, for instance. Is there a reason you've left
> this out?
>
> Cheers,
>
> Tom
>
> On 4 August 2017 at 18:47, Dong Lin  wrote:
>
> > Hey Ismael,
> >
> > Thanks for the comments! Here are my answers:
> >
> > 1. Yes it has been considered. Here are the reasons why we don't do it
> > through controller.
> >
> > - There can be use-cases where we only want to rebalance the load of log
> > directories on a given broker. It seems unnecessary to go through
> > controller in this case.
> >
> >  - If controller is responsible for sending ChangeReplicaDirRequest, and
> if
> > the user-specified log directory is either invalid or offline, then
> > controller probably needs a way to tell user that the partition
> > reassignment has failed. We currently don't have a way to do this since
> > kafka-reassign-partition.sh simply creates the reassignment znode without
> > waiting for response. I am not sure that is a good solution to this.
> >
> > - If controller is responsible for sending ChangeReplicaDirRequest, the
> > controller logic would be more complicated because controller needs to
> > first send ChangeReplicaRequest so that the broker memorize the partition
> > -> log directory mapping, send LeaderAndIsrRequest, and keep sending
> > ChangeReplicaDirRequest (just in case broker restarted) until replica is
> > created. Note that the last step needs repeat and timeout as the proposed
> > in the KIP-113.
> >
> > Overall I think this adds quite a bit complexity to controller and we
> > probably want to do this only if there is strong clear of doing so.
> > Currently in KIP-113 the kafka-reassign-partitions.sh is responsible for
> > sending ChangeReplicaDirRequest with repeat and provides error to user if
> > it either fails or timeout. It seems to be much simpler and user
> shouldn't
> > care whether it is done through controller.
> >
> > And thanks for the suggestion. I will add this to the Rejected
> Alternative
> > Section in the KIP-113.
> >
> > 2) I think user needs to be able to specify different log directories for
> > the replicas of the same partition in order to rebalance load across log
> > directories of all brokers. I am not sure I understand the question. Can
> > you explain a bit more why "that the log directory has to be the same for
> > all replicas of a given partition"?
> >
> > 3) Good point. I think the alterReplicaDir is a better than
> > changeReplicaDir for the reason you provided. I will also update names of
> > the request/response as well in the KIP.
> >
> >
> > Thanks,
> > Dong
> >
> > On Fri, Aug 4, 2017 at 9:49 AM, Ismael Juma  wrote:
> >
> > > Thanks Dong. I have a few initial questions, sorry if I it has been
> > > discussed and I missed it.
> > >
> > > 1. The KIP suggests that the reassignment tool is responsible for
> sending
> > > the ChangeReplicaDirRequests to the relevant brokers. I had imagined
> that
> > > this would be done by the Controller, like the rest of the reassignment
> > > process. Was this considered? If so, it would be good to include the
> > > details of why it was rejected in the "Rejected Alternatives" section.
> > >
> > > 2. The reassignment JSON format was extended so that one can choose the
> > log
> > > directory for a partition. This means that the log directory has to be
> > the
> > > same for all replicas of a given partition. The alternative would be
> for
> > > the log dir to be assignable for each replica. Similar to the other
> > > question, it would be good to have a section in "Rejected Alternatives"
> > for
> > > this approach. It's generally very helpful to have more information on
> > the
> > > rationale for the design choices that were made and rejected.
> > >
> > > 3. Should changeReplicaDir be alterReplicaDir? We have used `alter` for
> > > other methods.
> > >
> > > Thanks,
> > > Ismael
> > >
> > >
> > >
> > >
> > > On Fri, Aug 4, 2017 at 5:37 AM, Dong Lin  wrote:
> > >
> > > > Hi all,
> > > >
> > > > I realized that we need new API in AdminClient in order to use the
> new
> > > > request/response added in KIP-113. Since this is required by
> KIP-

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

[Sumant Tambe]

Please see the replies inline.

If we are going to have a separate configuration for expiry, I prefer my
> proposal of max.message.delivery.wait.ms and its semantics.
>
OK. I hope others will voice their preference too.


>
> However, one thing which has not come out of the JIRA discussion is the
> actual use cases for batch expiry.

There are two usecases I can think of for batch expiry mechanism
irrespective of how we try to bound the time (batch.expiry.ms or
max.message.delivery.wait.ms). Let's call it X.

1. A real-time app (e.g., periodic healthcheck producer, temperature sensor
producer) has a soft upper bound on both message delivery and failure
notification of message delivery. In both cases, it wants to know. Such an
app does not close the producer on the first error reported (due to batch
expiry) because there's data lined up right behind. It's ok to lose a few
samples of temperature measurement (IoT scenario). So it simply drops it
and moves on. May be when drop rate is like 70% it would close it. Such an
app may use acks=0. In this case, X will have some value in single digit
minutes. But X=MAX_LONG is not suitable.

2. Today we run KMM in Linkedin as if batch.expiry==MAX_LONG. We expire
under the condition: (!muted.contains(tp) && (isMetadataStale ||
cluster.leaderFor(tp) == null)) In essence, as long as the partition is
making progress (even if it's a trickle), the producer keeps on going.
We've other internal systems to detect whether a pipeline is making
*sufficient* progress or not. We're not dependent on the producer to tell
us that it's not making progress on a certain partition.

This is less than ideal though. We would be happy to configure
batch.expiry.ms to 1800,000 or so and upon notification of expiry restart
the process and what not. It can tell also tell us which specific
partitions of a specific topic is falling behind. We achieve a similar
effect via alternative mechanisms.

In the absence of a general out-of-band mechanism for discovering slowness
(or nonprogress), KIP-91 is an attempt to allow the producer to report
non-progress without using request.timeout.ms. Hence batch.expiry.ms.


> Also, the KIP document states the
> following:
>
> *The per message timeout is easy to compute - linger.ms
> >  + (retries + 1) * request.timeout.ms
> > ". *This is false.
>

> Why is the statement false? Doesn't that provide an accurate upperbound on
> the timeout for a produce request today?
>
The KIP-91 write-up describes the reasons why. Just reiterating the reason:
"the condition that if the metadata for a partition is known then we do not
expire its batches even if they are ready".  Do you not agree with the
explanation? If not, what part?

>
> Another point: the kip document keeps mentioning that the current timeouts
> are not intuitive, but for whom? In general, batch expiry as a notion is
> not intuitive and I am not sure the new settings change that fact.
>
Yeah, that's subjective.


>
> In this spirit, it might make sense to clarify the use case that motivates
> this additional setting. For instance, with this new configuration, how
> would your existing application handle a batch expired exception?

Again, a real-time app would just move on. KMM would halt. Any
order-sensitive app which needs to provide durability guarantees would
halt.


> How is it
> different from the way it handles the exception today?

It's not about how the existing applications will change their behavior.
It's about controlling *when*.


> Is the expiry
> exception a proxy for another piece of information like 'partition X is
> unavailable'?
>
Intriguing thought. If BatchExpiredException extends TimeoutException and
includes some context, such as TopicPartition, broker-id, an app may
provide differentiated service based on a topic name or availability zone
of a broker-id. KIP-91 does not propose anything like that. It's a very
niche usecase though.

Regards,
Sumant

>
>
>
>
>
> On Thu, Aug 3, 2017 at 4:35 PM, Sumant Tambe  wrote:
>
> > I don't want to list the alternatives in the JIRA as rejected just yet
> > because they are still being discussed. I would encourage the respective
> > proposers to do that. It's a wiki after all.
> >
> > As per my current understanding, there are two alternatives being
> proposed.
> > The original kip-91 approach #1 and #2 from Apurva. Apurva, correct me if
> > I'm wrong.
> >
> > #1. The batch.expiry.ms proposal: In this proposal the config is meant
> to
> > control ONLY the accumulator timeout. See the second diagram in kip-91.
> The
> > question "would the the clock for batch expiry be reset every time the
> > batch is requeued after failure?" does not arise here. There's no
> automatic
> > reenque. An application calls send again if it needs to in response to an
> > expired batch notification.
> >
> > #2. The max.message.delivery.wait.ms proposal: From Apurva's comment:
> > "...  if `T + max.message.delivery.wait.ms` has ela

[GitHub] kafka pull request #3638: MINOR: Fix missing wait_until import from bad cher...

[ewencp]

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/3638

MINOR: Fix missing wait_until import from bad cherry-pick



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka missing-wait-import

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3638.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3638


commit b335033bf551edb42f931a0698c6b2a1a01d3bd0
Author: Ewen Cheslack-Postava 
Date:   2017-08-07T15:39:26Z

MINOR: Fix missing wait_until import from bad cherry-pick




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

[Tom Bentley]

Hi,

I've updated this KIP slightly, to clarify a couple of points.

One thing in particular that I would like feedback on is what authorization
should be required for triggering the election of the preferred leader?

Another thing would be whether the request and responses should be grouped
by topic name. This would make for smaller messages when triggering
elections for multiple partitions of the same topic.

I'd be grateful for any feedback you may have.

Cheers,

Tom

On 2 August 2017 at 18:34, Tom Bentley  wrote:

> In a similar vein to KIP-179 I've created KIP-183 (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-183+-+Change+
> PreferredReplicaLeaderElectionCommand+to+use+AdminClient) which is about
> deprecating the --zookeeper option to kafka-preferred-replica-election.sh
> and replacing it with an option which would use a new AdminClient-based API.
>
> As it stands the KIP is focussed on simply moving the existing
> functionality behind the AdminClient.
>
> I'd be grateful for any feedback people may have on this.
>
> Thanks,
>
> Tom
>

[GitHub] kafka pull request #3434: KAFKA-5516: Formatting verifiable producer/consume...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3434


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3637: KAFKA-4879 KafkaConsumer.position may hang forever...

[baluchicken]

GitHub user baluchicken opened a pull request:

https://github.com/apache/kafka/pull/3637

KAFKA-4879 KafkaConsumer.position may hang forever when deleting a topic

@guozhangwang  I created this PR but it lacks the test. I also has the test 
but I wanted to ask which Test file should be the best candidate for this?
Also there is a TODO line which I created because that method already has a 
timeout parameter, what do you think how can we proceed there?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/baluchicken/kafka-1 KAFKA-4879

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3637.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3637


commit 211b2118b0ec5fd7ff5b499ef17c92b0d31e76e3
Author: Balint Molnar 
Date:   2017-08-07T14:54:34Z

KAFKA-4879 KafkaConsumer.position may hang forever when deleting a topic




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3636: KAFKA-5684: KStreamPrintProcessor as customized KS...

[ppatierno]

GitHub user ppatierno opened a pull request:

https://github.com/apache/kafka/pull/3636

KAFKA-5684: KStreamPrintProcessor as customized KStreamPeekProcessor

This PR is intended for having KStreamPrint derived from KStreamPeek and 
avoiding the "instance of" check on byte[ ] every process call.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ppatierno/kafka kafka-5684

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3636.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3636


commit a516a08bf22cdd53feb4ada50c73dc5a08715b74
Author: Paolo Patierno 
Date:   2017-08-07T12:55:16Z

Refactored KStreamPrint as derived from KStreamPeek
Removed the "instance of" check for byte[] in every KStreamPrint process, 
it's up to a default mapper now
Updated KStreamPrint tests adapting to the new internal structure for 
KStreamPrint and PrintForeachAction




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

[Ismael Juma]

Hi Dong,

Thanks for the explanation. Comments inline.

On Fri, Aug 4, 2017 at 6:47 PM, Dong Lin  wrote:

> 1. Yes it has been considered. Here are the reasons why we don't do it
> through controller.
>
> - There can be use-cases where we only want to rebalance the load of log
> directories on a given broker. It seems unnecessary to go through
> controller in this case.
>

Even though this is true, not sure how common it will be.

 - If controller is responsible for sending ChangeReplicaDirRequest, and if
> the user-specified log directory is either invalid or offline, then
> controller probably needs a way to tell user that the partition
> reassignment has failed. We currently don't have a way to do this since
> kafka-reassign-partition.sh simply creates the reassignment znode without
> waiting for response. I am not sure that is a good solution to this.
>

Since the JSON is provided by the user, we would ideally validate its
contents before storing it. Why are the log directories different than the
other information in the JSON?

- If controller is responsible for sending ChangeReplicaDirRequest, the
> controller logic would be more complicated because controller needs to
> first send ChangeReplicaRequest so that the broker memorize the partition
> -> log directory mapping, send LeaderAndIsrRequest, and keep sending
> ChangeReplicaDirRequest (just in case broker restarted) until replica is
> created. Note that the last step needs repeat and timeout as the proposed
> in the KIP-113.
>
> Overall I think this adds quite a bit complexity to controller and we
> probably want to do this only if there is strong clear of doing so.
> Currently in KIP-113 the kafka-reassign-partitions.sh is responsible for
> sending ChangeReplicaDirRequest with repeat and provides error to user if
> it either fails or timeout. It seems to be much simpler and user shouldn't
> care whether it is done through controller.
>

If I understand correctly, the logic is the same in both cases, it's just a
question of where it lives. The advantage of placing it in the Controller
is that the whole reassignment logic is in one place (instead of split
between the tool and the Controller). That seems easier to reason about.

Also, how do you think things would work in the context of KIP-179? Would
the tool still invoke these requests or would it be done by the broker
receiving the alterTopics/reassignPartitions protocol call?

And thanks for the suggestion. I will add this to the Rejected Alternative
> Section in the KIP-113.
>
> 2) I think user needs to be able to specify different log directories for
> the replicas of the same partition in order to rebalance load across log
> directories of all brokers. I am not sure I understand the question. Can
> you explain a bit more why "that the log directory has to be the same for
> all replicas of a given partition"?


I think I misunderstood the schema. The KIP has the following example:

"partitions" : [
{
  "topic" : str,
  "partition" : int,
  "replicas" : [int],
  "log_dirs" : [str]<-- NEW. A log directory can be either "any",
or a valid absolute path that begins with '/'. This is an optional filed.
It is treated as an array of "any" if this field is not explicitly
specified in the json file.
},
...
  ]

Is it right that `log_dirs` is an array in the same order as `replicas`?
That's a bit obscure and we should document it more clearly. Did we discard
the option of a more readable schema (i.e. a JSON object mapping a replica
id to a log dir) due to efficiency concerns? It would be good to include
that in the KIP.

Thanks,
Ismael

[GitHub] kafka pull request #3635: KAFKA-3417: Quote client-id and wrap reporter call...

[mimaison]

GitHub user mimaison opened a pull request:

https://github.com/apache/kafka/pull/3635

KAFKA-3417: Quote client-id and wrap reporter calls in try/catch blocks



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mimaison/kafka KAFKA-3417

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3635.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3635


commit ec36fa8052ea60411c77ed82c4b3d7b83fb1f4d3
Author: Mickael Maison 
Date:   2017-08-07T11:51:49Z

KAFKA-3417: Quote client-id and wrap reporter calls in try/catch blocks




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3634: MINOR: Deprecate LogConfig.Compact

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/3634

MINOR: Deprecate LogConfig.Compact

It actually refers to the `delete` cleanup policy.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka fix-misleading-compact-log-config

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3634.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3634


commit fba45066f843f6f65d871e82a13b393f82315d1a
Author: Ismael Juma 
Date:   2017-08-07T11:34:19Z

MINOR: Deprecate LogConfig.Compact

It actually refers to the `delete` cleanup policy.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

[Tom Bentley]

Hi Dong,

Your comments on KIP-179 prompted me to look at KIP-113, and I have a
question:

AFAICS the DescribeDirsResponse (via ReplicaInfo) can be used to get the
size of a partition on a disk, but I don't see a mechanism for knowing the
total capacity of a disk (and/or the free capacity of a disk). That would
be very useful information to have to help figure out that certain
assignments are impossible, for instance. Is there a reason you've left
this out?

Cheers,

Tom

On 4 August 2017 at 18:47, Dong Lin  wrote:

> Hey Ismael,
>
> Thanks for the comments! Here are my answers:
>
> 1. Yes it has been considered. Here are the reasons why we don't do it
> through controller.
>
> - There can be use-cases where we only want to rebalance the load of log
> directories on a given broker. It seems unnecessary to go through
> controller in this case.
>
>  - If controller is responsible for sending ChangeReplicaDirRequest, and if
> the user-specified log directory is either invalid or offline, then
> controller probably needs a way to tell user that the partition
> reassignment has failed. We currently don't have a way to do this since
> kafka-reassign-partition.sh simply creates the reassignment znode without
> waiting for response. I am not sure that is a good solution to this.
>
> - If controller is responsible for sending ChangeReplicaDirRequest, the
> controller logic would be more complicated because controller needs to
> first send ChangeReplicaRequest so that the broker memorize the partition
> -> log directory mapping, send LeaderAndIsrRequest, and keep sending
> ChangeReplicaDirRequest (just in case broker restarted) until replica is
> created. Note that the last step needs repeat and timeout as the proposed
> in the KIP-113.
>
> Overall I think this adds quite a bit complexity to controller and we
> probably want to do this only if there is strong clear of doing so.
> Currently in KIP-113 the kafka-reassign-partitions.sh is responsible for
> sending ChangeReplicaDirRequest with repeat and provides error to user if
> it either fails or timeout. It seems to be much simpler and user shouldn't
> care whether it is done through controller.
>
> And thanks for the suggestion. I will add this to the Rejected Alternative
> Section in the KIP-113.
>
> 2) I think user needs to be able to specify different log directories for
> the replicas of the same partition in order to rebalance load across log
> directories of all brokers. I am not sure I understand the question. Can
> you explain a bit more why "that the log directory has to be the same for
> all replicas of a given partition"?
>
> 3) Good point. I think the alterReplicaDir is a better than
> changeReplicaDir for the reason you provided. I will also update names of
> the request/response as well in the KIP.
>
>
> Thanks,
> Dong
>
> On Fri, Aug 4, 2017 at 9:49 AM, Ismael Juma  wrote:
>
> > Thanks Dong. I have a few initial questions, sorry if I it has been
> > discussed and I missed it.
> >
> > 1. The KIP suggests that the reassignment tool is responsible for sending
> > the ChangeReplicaDirRequests to the relevant brokers. I had imagined that
> > this would be done by the Controller, like the rest of the reassignment
> > process. Was this considered? If so, it would be good to include the
> > details of why it was rejected in the "Rejected Alternatives" section.
> >
> > 2. The reassignment JSON format was extended so that one can choose the
> log
> > directory for a partition. This means that the log directory has to be
> the
> > same for all replicas of a given partition. The alternative would be for
> > the log dir to be assignable for each replica. Similar to the other
> > question, it would be good to have a section in "Rejected Alternatives"
> for
> > this approach. It's generally very helpful to have more information on
> the
> > rationale for the design choices that were made and rejected.
> >
> > 3. Should changeReplicaDir be alterReplicaDir? We have used `alter` for
> > other methods.
> >
> > Thanks,
> > Ismael
> >
> >
> >
> >
> > On Fri, Aug 4, 2017 at 5:37 AM, Dong Lin  wrote:
> >
> > > Hi all,
> > >
> > > I realized that we need new API in AdminClient in order to use the new
> > > request/response added in KIP-113. Since this is required by KIP-113, I
> > > choose to add the new interface in this KIP instead of creating a new
> > KIP.
> > >
> > > The documentation of the new API in AdminClient can be found here
> > >  > > 113%3A+Support+replicas+movement+between+log+directories#KIP-113:
> > > Supportreplicasmovementbetweenlogdirectories-AdminClient>.
> > > Can you please review and comment if you have any concern?
> > >
> > > Thanks!
> > > Dong
> > >
> > >
> > >
> > > On Wed, Jul 12, 2017 at 10:44 AM, Dong Lin 
> wrote:
> > >
> > > > The protocol change has been updated in KIP-113
> > > >  > > 113%3A+Support+replicas+movement+between+l

[GitHub] kafka pull request #3633: MINOR: streams memory management docs

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/3633

MINOR: streams memory management docs

update streams memory management docs

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka mem-doc-011

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3633.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3633


commit b40bce07fe6eed4600dba1abf80faa52486f20f0
Author: Damian Guy 
Date:   2017-08-07T09:23:41Z

memory management docs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3604: MINOR: add memory management section to streams do...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3604


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-5707) Remove useless `--force` option for both TopicCommand and ConfigCommand

[huxihx (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

huxihx resolved KAFKA-5707.
---
Resolution: Not A Bug

For the sake of compatibility, just keep `--force` in both classes. Closed this 
jira then.

> Remove useless `--force` option for both TopicCommand and ConfigCommand
> ---
>
> Key: KAFKA-5707
> URL: https://issues.apache.org/jira/browse/KAFKA-5707
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 0.11.0.0
>Reporter: huxihx
>Assignee: huxihx
>Priority: Minor
>
> `TopicCommand` and `ConfigCommand` do expose an option named `--force` which 
> suppresses console prompts, but both classes do not actually use it. Should 
> remove it from the usage description.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #3632: KAFKA-5707: TopicCommand and ConfigCommand should ...

[huxihx]

Github user huxihx closed the pull request at:

https://github.com/apache/kafka/pull/3632


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-0.11.0-jdk7 #261

[Apache Jenkins Server]

See 


Changes:

[becket.qin] KAFKA-5700; Producer should not drop header information when 
splitting

--
[...truncated 972.91 KB...]
kafka.server.MultipleListenersWithDefaultJaasContextTest > testProduceConsume 
PASSED

kafka.server.EdgeCaseRequestTest > testInvalidApiVersionRequest STARTED

kafka.server.EdgeCaseRequestTest > testInvalidApiVersionRequest PASSED

kafka.server.EdgeCaseRequestTest > testMalformedHeaderRequest STARTED

kafka.server.EdgeCaseRequestTest > testMalformedHeaderRequest PASSED

kafka.server.EdgeCaseRequestTest > testProduceRequestWithNullClientId STARTED

kafka.server.EdgeCaseRequestTest > testProduceRequestWithNullClientId PASSED

kafka.server.EdgeCaseRequestTest > testInvalidApiKeyRequest STARTED

kafka.server.EdgeCaseRequestTest > testInvalidApiKeyRequest PASSED

kafka.server.EdgeCaseRequestTest > testHeaderOnlyRequest STARTED

kafka.server.EdgeCaseRequestTest > testHeaderOnlyRequest PASSED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK STARTED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK PASSED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot STARTED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot PASSED

kafka.server.ServerStartupTest > testConflictBrokerStartupWithSamePort STARTED

kafka.server.ServerStartupTest > testConflictBrokerStartupWithSamePort PASSED

kafka.server.ServerStartupTest > testConflictBrokerRegistration STARTED

kafka.server.ServerStartupTest > testConflictBrokerRegistration PASSED

kafka.server.ServerStartupTest > testBrokerSelfAware STARTED

kafka.server.ServerStartupTest > testBrokerSelfAware PASSED

kafka.server.ReplicationQuotaManagerTest > shouldThrottleOnlyDefinedReplicas 
STARTED

kafka.server.ReplicationQuotaManagerTest > shouldThrottleOnlyDefinedReplicas 
PASSED

kafka.server.ReplicationQuotaManagerTest > 
shouldSupportWildcardThrottledReplicas STARTED

kafka.server.ReplicationQuotaManagerTest > 
shouldSupportWildcardThrottledReplicas PASSED

kafka.server.ReplicationQuotaManagerTest > 
shouldExceedQuotaThenReturnBackBelowBoundAsTimePasses STARTED

kafka.server.ReplicationQuotaManagerTest > 
shouldExceedQuotaThenReturnBackBelowBoundAsTimePasses PASSED

kafka.server.KafkaMetricReporterClusterIdTest > testClusterIdPresent STARTED

kafka.server.KafkaMetricReporterClusterIdTest > testClusterIdPresent PASSED

kafka.server.ReplicationQuotasTest > 
shouldBootstrapTwoBrokersWithLeaderThrottle STARTED

kafka.server.ReplicationQuotasTest > 
shouldBootstrapTwoBrokersWithLeaderThrottle PASSED

kafka.server.ReplicationQuotasTest > shouldThrottleOldSegments STARTED

kafka.server.ReplicationQuotasTest > shouldThrottleOldSegments PASSED

kafka.server.ReplicationQuotasTest > 
shouldBootstrapTwoBrokersWithFollowerThrottle STARTED

kafka.server.ReplicationQuotasTest > 
shouldBootstrapTwoBrokersWithFollowerThrottle PASSED

kafka.server.AddPartitionsToTxnRequestTest > 
shouldReceiveOperationNotAttemptedWhenOtherPartitionHasError STARTED

kafka.server.AddPartitionsToTxnRequestTest > 
shouldReceiveOperationNotAttemptedWhenOtherPartitionHasError PASSED

kafka.server.AdvertiseBrokerTest > testBrokerAdvertiseHostNameAndPortToZK 
STARTED

kafka.server.AdvertiseBrokerTest > testBrokerAdvertiseHostNameAndPortToZK PASSED

unit.kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsShiftByHigherThanLatest STARTED

unit.kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsShiftByHigherThanLatest PASSED

unit.kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsShiftMinus 
STARTED

unit.kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsShiftMinus 
PASSED

unit.kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToCurrentOffset 
STARTED

unit.kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToCurrentOffset 
PASSED

unit.kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsToEarliestOnOneTopicAndPartition STARTED

unit.kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsToEarliestOnOneTopicAndPartition PASSED

unit.kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsByDurationToEarliest STARTED

unit.kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsByDurationToEarliest PASSED

unit.kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsToEarliestOnOneTopic STARTED

unit.kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsToEarliestOnOneTopic PASSED

unit.kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsNotExistingGroup STARTED

unit.kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsNotExistingGroup PASSED

unit.kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToZonedDateTime 
STARTED

unit.kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToZonedDateTime 
PASSED

unit.kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToEarliest 
STARTED

unit.kafka.admin.ResetConsumer

[jira] [Created] (KAFKA-5708) Update Jackson dependencies (from 2.8.5 to 2.9.x)

[JIRA]

Dejan Stojadinović created KAFKA-5708:
-

 Summary: Update Jackson dependencies (from 2.8.5 to 2.9.x)
 Key: KAFKA-5708
 URL: https://issues.apache.org/jira/browse/KAFKA-5708
 Project: Kafka
  Issue Type: Task
  Components: build
Reporter: Dejan Stojadinović
Priority: Blocker
 Fix For: 1.0.0


In addition to update: remove deprecated version forcing for 
'jackson-annotations'

*_Notes:_*
* wait until [Jackson 
2.9.1|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.9.1] is 
released (expected in September 2017)
* inspired by pull request: https://github.com/apache/kafka/pull/3631








--
This message was sent by Atlassian JIRA
(v6.4.14#64029)