Re: [DISCUSS] KIP-236 Interruptible Partition Reassignment

[Jun Rao]

Hi, Tom,

Thanks for the updated KIP. A few more comments below.

10. The proposal now stores the reassignment for all partitions in
/admin/reassignment_requests/request_xxx. If the number of reassigned
partitions is larger, the ZK write may hit the default 1MB limit and fail.
An alternative approach is to have the reassignment requester first write
the new assignment for each partition under
/admin/reassignments/$topic/$partition and then write
/admin/reassignment_requests/request_xxx with an empty value. The
controller can then read all values under /admin/reassignments.

11. The improvement you suggested in onPartitionReassignment() sounds good
at the high level. The computation of those dropped partitions seems a bit
complicated. Perhaps a simple approach is to drop the replicas not in the
original assignment and newest reassignment?

12. You brought up the need of remembering the original assignment. This
will be lost if the assignment is changed multiple times if we follow the
approach described in 10. One way is to store the original assignment in
/brokers/topics/[topic] as the following. When the final reassignment
completes, we can remove the original field.
{
  "version": 1,
  "partitions": {"0": [0, 1, 3] },
  "originals": {"0": [0, 1, 2] }
}

13. For resolving the conflict between /admin/reassign_partitions and
/admin/reassignments/$topic/$partition, perhaps it's more natural to just
let the assignment with a newer timestamp to override the older one?

14. Implementation wise, currently, we register a watcher of the isr path
of each partition being reassigned. This has the potential issue of
registering many listeners. An improvement could be just piggybacking on
the existing IsrChangeNotificationHandler, which only watches a single ZK
path and is triggered on a batch of isr changes. This is kind of orthogonal
to the KIP. However, if we are touching the reassignment logic, it may be
worth considering.

Thanks,

Jun

On Fri, Dec 15, 2017 at 10:17 AM, Tom Bentley  wrote:

> Just wanted to mention that I've started KIP-240, which builds on top of
> this one to provide an AdminClient API for listing and describing
> reassignments.
>
> On 15 December 2017 at 14:34, Tom Bentley  wrote:
>
> > > Should we seek to improve this algorithm in this KIP, or leave that as
> > a later optimisation?
> >
> > I've updated the KIP with a proposed algorithm.
> >
> >
> >
> >
> >
> >
> > On 14 December 2017 at 09:57, Tom Bentley  wrote:
> >
> >> Thanks Ted, now fixed.
> >>
> >> On 13 December 2017 at 18:38, Ted Yu  wrote:
> >>
> >>> Tom:
> >>> bq. create a znode /admin/reassignments/$topic-$partition
> >>>
> >>> Looks like the tree structure above should be:
> >>>
> >>> /admin/reassignments/$topic/$partition
> >>>
> >>> bq. The controller removes /admin/reassignment/$topic/$partition
> >>>
> >>> Note the lack of 's' for reassignment. It would be good to make
> zookeeper
> >>> paths consistent.
> >>>
> >>> Thanks
> >>>
> >>> On Wed, Dec 13, 2017 at 9:49 AM, Tom Bentley 
> >>> wrote:
> >>>
> >>> > Hi Jun and Ted,
> >>> >
> >>> > Jun, you're right that needing one watcher per reassigned partition
> >>> > presents a scalability problem, and using a separate notification
> path
> >>> > solves that. I also agree that it makes sense to prevent users from
> >>> using
> >>> > both methods on the same reassignment.
> >>> >
> >>> > Ted, naming the reassignments like mytopic-42 was simpler while I was
> >>> > proposing a watcher-per-reassignment (I'd have needed a child watcher
> >>> on
> >>> > /admin/reassignments and also on /admin/reassignments/mytopic). Using
> >>> the
> >>> > separate notification path means I don't need any watchers in the
> >>> > /admin/reassignments subtree, so switching to
> >>> /admin/reassignments/mytopic/
> >>> > 42
> >>> > would work, and avoid /admin/reassignments having a very large number
> >>> of
> >>> > child nodes. On the other hand it also means I have to create and
> >>> delete
> >>> > the topic nodes (e.g. /admin/reassignments/mytopic), which incurs the
> >>> cost
> >>> > of extra round trips to zookeeper. I suppose that since reassignment
> is
> >>> > generally a slow process it makes little difference if we increase
> the
> >>> > latency of the interactions with zookeeper.
> >>> >
> >>> > I have updated the KIP with these improvements, and a more detailed
> >>> > description of exactly how we would manage these znodes.
> >>> >
> >>> > Reading the algorithm in KafkaController.onPartitionReassignment(),
> it
> >>> > seems that it would be suboptimal for changing reassignments
> in-flight.
> >>> > Consider an initial assignment of [1,2], reassigned to [2,3] and then
> >>> > changed to [2,4]. Broker 3 will remain in the assigned replicas until
> >>> > broker 4 is in sync, even though 3 wasn't actually one of the
> original
> >>> > assigned replicas and is no longer a new assigned 

[GitHub] kafka-site issue #112: Migrate Streams Dev Guide content to AK

[derrickdoo]

Github user derrickdoo commented on the issue:

https://github.com/apache/kafka-site/pull/112
  
@joel-hamill just took a look at your branch. Content looks good, but I'd 
suggest adding that horizontal subnav on the developer guide page for 
consistency and make sure users can easily jump around.

Here's the page I'm looking at:

![image](https://user-images.githubusercontent.com/271961/34065592-185b40c8-e1b8-11e7-93a3-b93198c6053c.png)



---

[GitHub] kafka pull request #4335: MINOR: improve JavaDocs for KafkaStreams, KafkaPro...

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/4335

MINOR: improve JavaDocs for KafkaStreams, KafkaProducer, KafkaConsumer



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka 
minor-improve-KafkaStreams-javadocs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4335.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4335


commit 280580ec454c0a1b3e63a181bd153d0ad8edff2b
Author: Matthias J. Sax 
Date:   2017-12-15T23:55:11Z

MINOR: improve JavaDocs for KafkaStreams, KafkaProducer, KafkaConsumer




---

[GitHub] kafka pull request #4334: [KAFKA-6265] GlobalKTable missing #queryableStoreN...

[ConcurrencyPractitioner]

GitHub user ConcurrencyPractitioner opened a pull request:

https://github.com/apache/kafka/pull/4334

[KAFKA-6265] GlobalKTable missing #queryableStoreName()




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ConcurrencyPractitioner/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4334.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4334


commit 9264592828fe7bc38c660588c07526de17a5b761
Author: RichardYuSTUG 
Date:   2017-12-15T23:40:54Z

[KAFKA-6265] GlobalKTable missing #queryableStoreName()




---

[GitHub] kafka pull request #4333: MINOR: Use TopicPartition in ConsumerGroupCommand ...

[vahidhashemian]

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/4333

MINOR: Use TopicPartition in ConsumerGroupCommand instead of 
TopicAndPartition where possible


### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation 
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka 
minor/replace_topicandpartition_with_topicpartition_in_consumergroupcommand

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4333.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4333


commit 98e242368c0d677e0154f5f4f4a3d522b83b6b08
Author: Vahid Hashemian 
Date:   2017-12-15T23:28:12Z

MINOR: Use TopicPartition in ConsumerGroupCommand instead of 
TopicAndPartition where possible




---

[GitHub] kafka pull request #4332: KAFKA-6302: Improve AdmintClient JavaDocs

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/4332

KAFKA-6302: Improve AdmintClient JavaDocs




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka 
kafka-6302-update-admin-client-javadoc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4332.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4332


commit c07ab77b7897fd1036b8111a2353e77837fe0f4b
Author: Matthias J. Sax 
Date:   2017-12-15T23:20:05Z

KAFKA-6302: Improve AdmintClient JavaDocs




---

[GitHub] kafka pull request #3990: KAFKA-5473: handle ZK session expiration properly ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3990


---

[GitHub] kafka pull request #4331: [WIP] MINOR: Improve Join integration test coverag...

[guozhangwang]

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/4331

[WIP] MINOR: Improve Join integration test coverage, PART I

1. Merge `KStreamRepartitionJoinTest.java` into `JoinIntegrationTest.java` 
with augmented stream-stream join.

2. Make the JoinIntegrationTest `Parameterized` with and without caching.

### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation 
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka 
KMinor-join-integration-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4331.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4331


commit c1e00d0c5717a23cdf7252e48dcd6e49dd0ad810
Author: Guozhang Wang 
Date:   2017-11-30T01:01:17Z

ingested stream stream join

commit 71f2625746d4ce9d6297b8adc84d3df4eac435fb
Author: Guozhang Wang 
Date:   2017-12-15T22:22:07Z

Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into 
KMinor-join-integration-tests




---

[jira] [Created] (KAFKA-6375) Follower replicas can never catch up to be ISR due to creating ReplicaFetcherThread failed.

[Rong Tang (JIRA)]

Rong Tang created KAFKA-6375:


 Summary: Follower replicas can never catch up to be ISR due to 
creating ReplicaFetcherThread failed.
 Key: KAFKA-6375
 URL: https://issues.apache.org/jira/browse/KAFKA-6375
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.10.2.0
 Environment: Windows,  23 brokers KafkaCluster
Reporter: Rong Tang


Hi, I met with a case that in one broker, the out of sync replicas never catch 
up.
When the broker starts up, it receives LeaderAndISR requests from controller, 
which will call createFetcherThread, the thread creation failed, with 
exceptions below.

And then, there is no fetcher for these follower replicas, and it is out of 
sync forever. Unless, later, it receives LeaderAndISR requests that has higher 
leader EPOCH. 

Restart the broker can mitigate the issue.

I have 2 questions.  
First, Why NEW ReplicaFetcherThread failed?
*Second, shouldn't Kafka do something to fail over, instead of letting the 
broker in abnormal state.*

It is a 23 brokers Kafka cluster running on Windows. each broker has 330 
replicas.

[2017-12-13 16:29:21,317] ERROR Error on broker 1000 while processing 
LeaderAndIsr request with correlationId 1 received from controller 427703487 
epoch 22 (state.change.logger)
org.apache.kafka.common.KafkaException: java.io.IOException: *Unable to 
establish loopback connection
at org.apache.kafka.common.network.Selector.(Selector.java:124)
at 
kafka.server.ReplicaFetcherThread.(ReplicaFetcherThread.scala:87)
at 
*kafka.server.ReplicaFetcherManager.createFetcherThread(ReplicaFetcherManager.scala:35)
at 
kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:83)
at 
kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:78)
at 
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at 
scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221)
at 
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
at 
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
at 
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at 
kafka.server.AbstractFetcherManager.addFetcherForPartitions(AbstractFetcherManager.scala:78)
at kafka.server.ReplicaManager.makeFollowers(ReplicaManager.scala:869)
at 
kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:689)
at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:149)
at kafka.server.KafkaApis.handle(KafkaApis.scala:83)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Unable to establish loopback connection
at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:94)
at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at sun.nio.ch.PipeImpl.(PipeImpl.java:171)
at 
sun.nio.ch.SelectorProviderImpl.openPipe(SelectorProviderImpl.java:50)
at java.nio.channels.Pipe.open(Pipe.java:155)
at sun.nio.ch.WindowsSelectorImpl.(WindowsSelectorImpl.java:127)
at 
sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvider.java:44)
at java.nio.channels.Selector.open(Selector.java:227)
at org.apache.kafka.common.network.Selector.(Selector.java:122)
... 16 more
Caused by: java.net.ConnectException: Connection timed out: connect
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:454)
at sun.nio.ch.Net.connect(Net.java:446)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
at java.nio.channels.SocketChannel.open(SocketChannel.java:189)
at 
sun.nio.ch.PipeImpl$Initializer$LoopbackConnector.run(PipeImpl.java:127)
at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:76)
... 25 more




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : kafka-trunk-jdk8 #2279

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

[Guozhang Wang]

Jan,

Thanks for the updated KIP, and the raised questions. Here are my thoughts
around the "back and forth mapper" approach on your wiki:

1) regarding the key-value types of KTableValueGetter, we do not
necessarily enforce its template K, V to be the same as its calling
Processor, although today in all implementations we happen to do so.
So I think it is ok to extend this internal implementation to allow getter
and forwarding with different types.

2) regarding the KTableProcessorSupplier enforcing to return the same
key/value types of its "KTableValueGetterSupplier view();" the key
observation to note is that "ProcessorSupplier" inside "KTableImpl" does not enforce to have the same key-value types of the KTable, i.e.
we can use a "ProcessorSupplier" inside the impl of a `KTable`. I think that should help getting around the issue.

3) About the alternative KTable::mapKeys(), I think the major issue is that
this mapKeys() cannot enforce users to always call it to get the
"non-combined" Key, and hence users may still need to consider the serde of
"CombinedKey" if they do not call mapKeys and then directly pipe it to the
output, while this approach enforce them to always "map" it before trying
to write it to anywhere externally exposable.

4) A very minor comment on the wiki page itself, about the "back and forth
mapper" section: the parameter names "customCombinedKey" and "combinedKey"
seems a bit hard to understand to normal users; should we consider renaming
them to something more understandable? For example, "outputKeyCombiner" and
"outputKeySpliter"?



Guozhang


On Thu, Dec 7, 2017 at 3:58 AM, Jan Filipiak 
wrote:

>
> On 05.12.2017 00:42, Matthias J. Sax wrote:
>
>> Jan,
>>
>> The KTableValueGetter thing is a valid point. I think we would need a
>> backwards mapper (or merge both into one and sacrifices lambdas?).
>> Another alternative would be, to drop the optimization and materialize
>> the KTable.operator() result... (not a great solution either). I am
>> personally fine with a backwards mapper (we should call it KeySplitter).
>>
>> 2. I am not sure if we can pull it of w/o said forth generic type in
 KTable (that I am in favour of btw)

>>> Not sure if I can follow here. I am personally not worried about the
>> number of generic types -- it's just to have a clear definition what
>> each passed parameter does.
>>
> I need to double check this again. Its good that we are open to introduce
> a new one
> I think it will not work currently as a KTableProcessorSupplier when asked
> for a
> ValueGetterSupplier it can only return a ValueGetter Supplier that has the
> same Keytype
> as the key it receives in the process method. Even though it would forward
> a different
> key type and therefore KTables key Type can't change. I am thinking how to
> pull this off but I see little chance
>
> But I am always in big favour of introducing the forth type OutputKey, it
> would become
> straight forward then. I hope you can follow.
>
> + It won't solves peoples problem having CombinedKey on the wire and not
>>> being able to inspect the topic with say there default tools.
>>>
>> I see your point, but do we not have this issue always? To make range
>> scan work, we need to serialize the prefix (K1) and suffix (K)
>> independently from each other. IMHO, it would be too much of a burden to
>> the user, to provide a single serialized for K0 that guaranteed the
>> ordering we need. Still, advanced user can provide custom Serde for the
>> changelog topic via `Joined` -- and they can serialize as they wish (ie,
>> get CombinedKey, convert internally to K0 and serialized -- but
>> this is an opt-in).
>>
>> I think, this actually aligns with what you are saying. However, I think
>> the #prefix() call is not the best idea. We can just use Serde for
>> this (if users overwrite CombinedKey-Serde, it must overwrite Serde
>> too and can return the proper perfix (or do I miss something?).
>>
> I can't follow. For the stock implementation user would get
> they wouldn't need prefix. Users had not to define it we can implement
> that ourself by just getting K1 Serde.
>
> But to Override with a custom Serde that prefix method is needed as an
> indicator if only a prefix or the full thing is to be rendered.
>
>
>>   - Id rather introduce KTable::mapKeys() or something (4th generic in
>>> Ktable?) than overloading. It is better SOCs wise.
>>>
>> What overload are you talking about? From my understanding, we want to
>> add one single method (or maybe one for inner,left,outter each), but I
>> don't see any overloads atm?
>>
> The back and forth mapper would get an overload
>
>>
>> Also, `KTable.mapKeys()` would have the issue, that one could create an
>> invalid KTable with key collisions. I would rather shield users to shoot
>> themselves in the foot.
>>
> This mapkeys would not be used to remove the actual values but to get rid
> of the CombinedKey-type.
> 

Re: [VOTE] KIP-239 Add queryableStoreName() to GlobalKTable

[Bill Bejeck]

+1

On Fri, Dec 15, 2017 at 3:57 PM, Guozhang Wang  wrote:

> +1 (binding), thanks!
>
> On Fri, Dec 15, 2017 at 11:56 AM, Ted Yu  wrote:
>
> > Hi,
> > Here is the discussion thread:
> >
> > http://search-hadoop.com/m/Kafka/uyzND12QnH514pPO9?subj=
> > Re+DISCUSS+KIP+239+Add+queryableStoreName+to+GlobalKTable
> >
> > Please vote on this KIP.
> >
> > Thanks
> >
>
>
>
> --
> -- Guozhang
>

[jira] [Created] (KAFKA-6374) Constant Consumer Errors after replacing a broker

[Aravind Velamur Srinivasan (JIRA)]

Aravind Velamur Srinivasan created KAFKA-6374:
-

 Summary: Constant Consumer Errors after replacing a broker
 Key: KAFKA-6374
 URL: https://issues.apache.org/jira/browse/KAFKA-6374
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 0.10.2.1
 Environment: OS: linux
Broker Instances: EC2 (r4.xlarge)
Storage: EBS (HDD st1 - 16T size)
Client: golang (sarama and sarama-cluster libraries)
Cluster Size: 5 nodes
Kafka Version: 0.10.2.1
ZooKeeper: 3 nodes (separate from the brokers)
Reporter: Aravind Velamur Srinivasan


We had to replace one of the brokers for maintenance reasons. We did the 
following to replace the broker:
(1) Gracefully stop the Kafka broker (id: 48)
(2) Make sure producers/consumers were fine (the consumers coordinated by this 
broker now were managed by another broker and things were fine)
(3) Spin up a new instance with the same IP
(4) Make sure the new instance's config is the same as old with the same broker 
ID.
(5) Bring the new one back up.

It took ~35 to 40 mins to do this. But once the broker came back up, the 
consumer groups coordinated by this broker were getting constant errors that 
this CG is not coordinated by this broker for nearly 30 to 40 mins until i 
stopped the broker again.

Looks like the metadata kept returning that the coordinator for this CG is the 
same old broker (id 48) even after the client kept on asking for the 
coordinator.

(1) Are there any known issues/recent fixes for this?
(2) Why didn't the metadata refresh? Any ideas on what could be happening?

We were constantly getting errors when trying to fetch the coordinator like 
this:
'
sarama-logger : client/coordinator requesting coordinator for consumergroup 
from  (some other broker)
sarama-logger : client/coordinator coordinator for consumergroup is #48
kafka- Error: kafka server: Request was for a consumer group that is not 
coordinated by this broker.
'

In the kafka broker saw lots of errors like this:
'
[2017-12-13 00:38:49,559] ERROR [ReplicaFetcherThread-0-48], Error for 
partition [__consumer_offsets,37] to broker 
48:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server 
is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
'

Is it running into the stale metadata like this:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design#Kafka0.9ConsumerRewriteDesign-Co-ordinatorfailoverorconnectionlosstotheco-ordinator



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] KIP-239 Add queryableStoreName() to GlobalKTable

[Guozhang Wang]

+1 (binding), thanks!

On Fri, Dec 15, 2017 at 11:56 AM, Ted Yu  wrote:

> Hi,
> Here is the discussion thread:
>
> http://search-hadoop.com/m/Kafka/uyzND12QnH514pPO9?subj=
> Re+DISCUSS+KIP+239+Add+queryableStoreName+to+GlobalKTable
>
> Please vote on this KIP.
>
> Thanks
>



-- 
-- Guozhang

Re: [DISCUSS] KIP-231: Improve the Required ACL of ListGroups API

[Vahid S Hashemian]

If there are no other feedback or suggestion on this KIP, I'll start a 
vote early next week.

Thanks.
--Vahid



From:   "Vahid S Hashemian" 
To: dev@kafka.apache.org
Date:   11/29/2017 03:18 PM
Subject:Re: [DISCUSS] KIP-231: Improve the Required ACL of 
ListGroups API



Completing the subject line :)



From:   "Vahid S Hashemian" 
To: dev 
Date:   11/29/2017 03:17 PM
Subject:[DISCUSS] KIP-231:



Hi everyone,

I started KIP-231 to propose a small change to the required ACL of 
ListGroups API (in response to KAFKA-5638): 
https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D231-253A-2BImprove-2Bthe-2BRequired-2BACL-2Bof-2BListGroups-2BAPI=DwIFAg=jf_iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=XjHVTsIl7t-z0NBesB0U-ptMMm6mmpy3UqS8TjJM5yM=eu378oaLvC0Wzbfcz15Rwo4nqdrO11ENLK6v9Kq9Z6w=


Your feedback and suggestions are welcome!

Thanks.
--Vahid

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

[Guozhang Wang]

Regarding the record context inheritance: I agree it may be a better idea
for now to drop the information when we cannot come up with a consensus
about how the record context should be inherited. Like Bill I was a bit
worried about the lacking of such data lineage information for trouble
shooting in operations or debugging in coding; but I think we can still try
to come up with better solutions in the future by extending the current
protocol, than coming up with something that we realized that we need to
change in the future.

Regarding the table / filter question: I agree with Jan that we could
consider update the builder so that we will push down the filter earlier
than KTable source that materialized the store; on the other hand, I think
Matthias' point is that even doing this does not completely exclude the
scenarios that you'd have the old/new pairs in your Tables, for example,
consider:

table1 = stream1.groupBy(...).aggregate(...)
table2 = table1.filter(..., Materialized.as(...))

In this case table2 is filtering on table1 which is not read from the
source, and hence it already outputs the old/new pairs already, so we still
need to consider how to handle it.


So I'd suggest the following execution plan towards KIP-159:

1) revisit our record context (topic, partition, offset, timestamp)
protocols that is used at the DSL layer, make it clear at which high-level
operators we should apply certain inheritance rule, and which others we
should drop such information.
1.1) modify the lower-level PAPI that DSL leverages, to allow the
caller (DSL) to modify the record context (note that today for lower-level
API, the record context is always passed through when forwarding to the
next processor node)
2) at the same time, consider optimizing the source KTable filter cases (I
think we already have certain JIRA tickets for this) so that the filter
operator is pushed early than the KTABLESOURCE node where materialization
happens.
3) after 1) is done, come back to KIP-159 and add the proposed APIs.


Guozhang


On Thu, Dec 7, 2017 at 12:27 PM, Jan Filipiak 
wrote:

> Thank you Bill,
>
> I think this is reasonable. Do you have any suggestion
> for handling oldValues in cases like
>
> builder.table().filter(RichPredicate).join()
>
> where we process a Change with old and new value and dont have a record
> context for old.
>
> my suggestion would be that instead of
>
> SOURCE -> KTABLESOURCE -> KTABLEFILTER -> JOIN -> SINK
>
> we build
>
> SOURCE  -> KTABLEFILTER ->  KTABLESOURCE -> JOIN -> SINK
>
> We should build a topology like this from the beginning and not have
> an optimisation phase afterwards.
>
> Any opinions?
>
> Best Jan
>
>
>
>
> On 05.12.2017 17:34, Bill Bejeck wrote:
>
>> Matthias,
>>
>> Overall I agree with what you've presented here.
>>
>> Initially, I was hesitant to remove information from the context of the
>> result records (Joins or Aggregations) with the thought that when there
>> are
>> unexpected results, the source information would be useful for tracing
>> back
>> where the error could have occurred.  But in the case of Joins and
>> Aggregations, the amount of data needed to do meaningful analysis could be
>> too much. For example, a join result could come from two topics so you'd
>> need to keep both original topic names, offsets, etc. (plus the broker
>> could have deleted the records in the interim so even having offset could
>> provide nothing).
>>
>> I'm bit long winded here, but I've come full circle to your original
>> proposal that since Joins and Aggregations produce fundamentally new
>> types,
>> we drop the corresponding information from the context even in the case of
>> single topic aggregations.
>>
>> Thanks,
>> Bill
>>
>> On Mon, Dec 4, 2017 at 7:02 PM, Matthias J. Sax 
>> wrote:
>>
>> I agree with Guozhang that just exposing meta data at the source level
>>> might not provide too much value. Furthermore, for timestamps we do
>>> already have a well defined contract and we should exploit it:
>>> timestamps can always be provided in a meaningful way.
>>>
>>> Also, for simple operations like KStream-filter/map the contract is
>>> simple and we can just use it. Same for KTable-filter/map (for new
>>> values).
>>>
>>> For aggregations, join, and oldValue, I could just drop some information
>>> and return `null`/-1, if the result records has no semantically
>>> meaningful meta data.
>>>
>>> For example, for aggregations, we could preserve the partition (as all
>>> agg-input-records have the same partition). For single input topic
>>> aggregation (what I guess is the most prominent case), we can also carry
>>> over the topic name (would be a internal repartitioning topic name
>>> often). Offsets don't have any semantic interpretation IMHO and we could
>>> return -1.
>>>
>>> For joins, we could keep the partition information. Topic and offset are
>>> both unknown/invalid for the output record IMHO.
>>>
>>> For the oldValue case, 

Re: [VOTE] KIP-239 Add queryableStoreName() to GlobalKTable

[Matthias J. Sax]

+1

On 12/15/17 11:56 AM, Ted Yu wrote:
> Hi,
> Here is the discussion thread:
> 
> http://search-hadoop.com/m/Kafka/uyzND12QnH514pPO9?subj=Re+DISCUSS+KIP+239+Add+queryableStoreName+to+GlobalKTable
> 
> Please vote on this KIP.
> 
> Thanks
> 



signature.asc
Description: OpenPGP digital signature

Jenkins build is back to normal : kafka-trunk-jdk7 #3041

[Apache Jenkins Server]

See 

[VOTE] KIP-239 Add queryableStoreName() to GlobalKTable

[Ted Yu]

Hi,
Here is the discussion thread:

http://search-hadoop.com/m/Kafka/uyzND12QnH514pPO9?subj=Re+DISCUSS+KIP+239+Add+queryableStoreName+to+GlobalKTable

Please vote on this KIP.

Thanks

[jira] [Created] (KAFKA-6373) Log end offset of input table changing during restore

[Charles Crain (JIRA)]

Charles Crain created KAFKA-6373:


 Summary: Log end offset of input table changing during restore
 Key: KAFKA-6373
 URL: https://issues.apache.org/jira/browse/KAFKA-6373
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 1.0.0
 Environment: Client 1.0.0, Brokers 1.0.0 with 1.0.0 message format and 
inter-broker protocol
Reporter: Charles Crain


I am receiving a confusing error from a Kafka Streams application.  Most of the 
time when I try to being up just a single replica of the task for the first 
time, I get this:

{noformat}
Detected a task that got migrated to another thread. This implies that this 
thread missed a rebalance and dropped out of the consumer group. Trying to 
rejoin the consumer group now.
org.apache.kafka.streams.errors.TaskMigratedException: Log end offset of [Name 
of Topic]-36 should not change while restoring: old end offset 37559, current 
offset 37561
{noformat}

The confusing thing is that [Name of Topic] is *not* a change log topic created 
by the stream app.  Rather it is a topic published from a completely different 
service.  And since that other service is publishing to that topic actively, of 
course the end offset is constantly changing.

Here is a rough view of my stream topology.  I'll call the topic that's showing 
up in the above error "ExternalTableTopic".


{noformat}
externalTable = table(ExternalTableTopic)

stream(ExternalStreamTopic)
  .leftJoin(externalTable, joiner)
  .aggregate(aggregator, SomeQueryableStoreName)
{noformat}


...and that's it.  If I take out the left join this appears not to happen.  Is 
it illegal to join a table to a stream if that table is being published from 
somewhere else?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-239 Add queryableStoreName() to GlobalKTable

[Guozhang Wang]

Ted,

Thanks for creating the KIP. I have made a pass over it and it looks good
to me.

Note that since our current KafkaStreams#store(storeName, ..) already
checks for global stores first with the given storeName (this is exactly
the queryableStoreName). Having this KIP would naturally allow users to get
the store name and is that to query the global state store with DSL.

So to me this should be a straight-forward API to add. I'm +1 on this.


Guozhang


On Tue, Dec 12, 2017 at 8:50 AM, Ted Yu  wrote:

> Hi,
> Please kindly provide feedback on the following KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 239+Add+queryableStoreName%28%29+to+GlobalKTable
>
> Thanks
>



-- 
-- Guozhang

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

[Colin McCabe]

On Fri, Dec 15, 2017, at 08:56, dan wrote:
> i didn't think it would be appropriate to return `num.partitions` as a
> Config for a Topic since that is not a valid config for that resource
> type.
> 
> yes, that is another option i have on the wiki in the Rejected
> Alternatives section (though, its really just an alternative until something 
> else
> passes voting :). we still have the issue with return type here, NewTopic has
> the info i want to return, but not necessarily the name i'd want...

Hi Dan,

Hmm.  I don't think NewTopic is really the right thing to return,
though.  NewTopic is structured so that it can be vague... you can
specify just replication factor, or a full assignment.  You can specify
some configuration keys, or no configuration keys at all.

NewTopic will probably grow more optional fields in the future.  But the
key thing is that NewTopic represents a request to create something, not
a description of something that actually exists.

> it could
> make sense for a createTopics() to return a TopicDescription and we could
> add the config/replication info to that class, but we'd then need to add
> all this info for all describeTopics() calls. not sure of the
> ramifications/overhead here.

Replication info is already part of TopicDescription, right?  There is a
list of TopicPartitionInfo there.

Config already encapsulates everything you would want to know about
topic configs, I think.

We don't really need to have a new type that encloses both descriptions
and configs.  We can just have

CreateTopicsResult {
  KafkaFuture description(String topic);
  KafkaFuture config(String topic);
  Map values();
  KafkaFuture all();
}

And then people who care about the config of the topic they just created
can call result.config("myTopicName").get(), and those who don't care
can ignore it.

This would be really nice... it lets you figure out what the partition
assignment and config actually are when you actually create a topic. 
Plus it extends the existing createTopics(validateOnly=true)
functionality to be even more useful.

best,
Colin


> 
> dan
> 
> On Wed, Dec 13, 2017 at 10:10 AM, Colin McCabe 
> wrote:
> 
> > On Wed, Dec 13, 2017, at 10:00, dan wrote:
> > > > Why not just return
> > > > org.apache.kafka.clients.admin.Config like describeConfigs does?
> > >
> > > brokers have a `num.partitions` config that does not map to a valid
> > > `Config` entry for a topic.
> >
> > Hi Dan,
> >
> > Sorry if I'm misunderstanding something, but why not map it to
> > num.partitions?
> >
> > >
> > > another added benefit to using `NewTopic` may be (future kip) having the
> > > cluster return the actual replica mappings it would create (i have no
> > > idea if this is actually possible)
> >
> > A better way of doing that would probably be extending
> > CreateTopicsRequest so that it returns partition assignment information
> > to the caller.  Then using validOnly = true to get this information.
> >
> > Actually, come to think of it, maybe we should be doing that for this
> > KIP too.  Why not have CreateTopicsRequest return the config that was
> > used, plus the partition assignment that was made?  We don't create
> > topics that often, so the extra space on the wire should not be a
> > concern.
> >
> > best,
> > Colin
> >
> > >
> > > dan
> > >
> > > On Wed, Dec 13, 2017 at 9:55 AM, Colin McCabe 
> > wrote:
> > >
> > > > On Tue, Dec 12, 2017, at 19:02, Ewen Cheslack-Postava wrote:
> > > > > re: API versions, I actually wasn't sure if we needed it or not. I'm
> > fine
> > > > > if people would prefer just bumping it, but I was actually curious
> > if we
> > > > > could get away without bumping it. I don't know the behavior of the
> > > > > broker code paths for this well enough to know what types of errors
> > those
> > > > > non-null assertions get converted into.
> > > >
> > > > There's no advantage to trying to keep the API version number the same,
> > > > though.  Since we have bidirectional client compatibility now, the
> > > > clients and the server will just negotiate whatever version they need.
> > > > New clients can still talk to older brokers that don't support this
> > > > feature.
> > > >
> > > > If you don't bump the API version, the best case scenario is that you
> > > > get a disconnect exception and the end-user is left confused about why.
> > > > The worse-case scenario is that you crash the broker (but probably not,
> > > > since you'd just get an NPE in serde, I think).  If you bump the
> > version
> > > > number, you can provide a proper UnsupportedVersionException when the
> > > > feature is not supported.
> > > >
> > > > > For the return type, NewTopic seems reasonable and kind of intuitive
> > --
> > > > > basically a description of the NewTopic you would get. The only
> > reason I
> > > > > would be wary of reusing it is that what we don't want people doing
> > is
> > > > > taking that and passing it directly into 

Re: Permission to create KIP

[Jun Rao]

Hi, Vincent,

Thanks for your interest. Just gave you the wiki permission.

Jun

On Fri, Dec 15, 2017 at 10:37 AM, Qiao Meng  wrote:

> Hi,
>
> Can I get the permission to create KIP? my user handle is
> "suanmeiguo", or using email "suanmei...@gmail.com".
>
> Thank you!
>
> -Vincent
>

Re: [VOTE] KIP-238: Expose Kafka cluster ID in Connect REST API

[Ted Yu]

+1

On Fri, Dec 15, 2017 at 10:49 AM, Ewen Cheslack-Postava 
wrote:

> Discussion seems to have tapered off, so I'd like to start the vote on
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 238%3A+Expose+Kafka+cluster+ID+in+Connect+REST+API
>
> Obviously +1 (binding) from me :)
>
> -Ewen
>

Re: [DISCUSS] KIP-240: AdminClient.listReassignments AdminClient.describeReassignments

[Ted Yu]

Please create corresponding JIRA.

For class Reassignment, it seems you forgot to include set of brokers.

For class DescribeReassignmentsResult:
public KafkaFuture reassignments();
the return value should be a Collection.


On Fri, Dec 15, 2017 at 10:16 AM, Tom Bentley  wrote:

> Hi,
>
> KIP-236 lays the foundations for AdminClient APIs to do with partition
> reassignment. I'd now like to start discussing KIP-240, which adds APIs to
> the AdminClient to list and describe the current reassignments.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-240%3A+AdminClient.
> listReassignments+AdminClient.describeReassignments
>
> Aside: I have fairly developed ideas for the API for starting a
> reassignment, but I intend to put that in a third KIP.
>
> Cheers,
>
> Tom
>

Re: [DISCUSS] KIP-240: AdminClient.listReassignments AdminClient.describeReassignments

[Steven Aerts]

Tom,


I think it would be useful to be able to subscribe yourself on updates of
reassignment changes.
Our internal kafka supervisor and monitoring tools are currently subscribed
to these changes in zookeeper so they can babysit our clusters.

I think it would be nice if we could receive these events through the
adminclient.
In the api proposal, you can only poll for changes.

No clue how difficult it would be to implement, maybe you can piggyback on
some version number in the repartition messages or on zookeeper.

This is just an idea, not a must have feature for me.  We can always poll
over
the proposed api.


Regards,


   Steven


Op vr 15 dec. 2017 om 19:16 schreef Tom Bentley :

> Hi,
>
> KIP-236 lays the foundations for AdminClient APIs to do with partition
> reassignment. I'd now like to start discussing KIP-240, which adds APIs to
> the AdminClient to list and describe the current reassignments.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-240%3A+AdminClient.listReassignments+AdminClient.describeReassignments
>
> Aside: I have fairly developed ideas for the API for starting a
> reassignment, but I intend to put that in a third KIP.
>
> Cheers,
>
> Tom
>

[VOTE] KIP-238: Expose Kafka cluster ID in Connect REST API

[Ewen Cheslack-Postava]

Discussion seems to have tapered off, so I'd like to start the vote on
https://cwiki.apache.org/confluence/display/KAFKA/KIP-238%3A+Expose+Kafka+cluster+ID+in+Connect+REST+API

Obviously +1 (binding) from me :)

-Ewen

Permission to create KIP

[Qiao Meng]

Hi,

Can I get the permission to create KIP? my user handle is
"suanmeiguo", or using email "suanmei...@gmail.com".

Thank you!

-Vincent

[jira] [Created] (KAFKA-6372) Trogdor should use LogContext for log messages

[Colin P. McCabe (JIRA)]

Colin P. McCabe created KAFKA-6372:
--

 Summary: Trogdor should use LogContext for log messages
 Key: KAFKA-6372
 URL: https://issues.apache.org/jira/browse/KAFKA-6372
 Project: Kafka
  Issue Type: Sub-task
Reporter: Colin P. McCabe
Assignee: Colin P. McCabe
Priority: Minor


Trogdor should use LogContext for log messages, rather than manually prefixing 
log messages with the context.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #4271: KAFKA-5526: Additional `--describe` views for Cons...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4271


---

[GitHub] kafka pull request #4330: [WIP] KAFKA-6359: KIP-236 interruptible reassignme...

[tombentley]

GitHub user tombentley opened a pull request:

https://github.com/apache/kafka/pull/4330

[WIP] KAFKA-6359: KIP-236 interruptible reassignments

This is a WIP for KIP-236. All the existing tests (via the 
`/admin/reassign_partitions` path) still pass, and I've added a couple of tests 
for the new path (via `/admin/reassignment_requests`), but it needs a lot more 
tests of the new path.

### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation 
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tombentley/kafka 
KIP-236-interruptible-reassignments

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4330.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4330


commit eb856a3009e5cc3d293305fe6a3add0d21267095
Author: Tom Bentley 
Date:   2017-12-06T15:18:17Z

Split /admin/reassign_partitions

commit 6114752becb964041da88e874ce289968c2d2abc
Author: Tom Bentley 
Date:   2017-12-07T09:40:42Z

KIP-236 wip

commit 5caf4e3b6e9fc5ff4be3755ed77057598438017d
Author: Tom Bentley 
Date:   2017-12-13T09:21:33Z

Jun's idea of using separate path for notification

commit 325569c1adfb220eaef3e6deb642d55fbab3d5ce
Author: Tom Bentley 
Date:   2017-12-13T13:22:04Z

Tidy up

commit 6c3a922fde6a2f1b0c6acca80f8c175948b165a9
Author: Tom Bentley 
Date:   2017-12-14T10:14:26Z

More work

commit bb9e3874590cc71825fea37a92f77b6b426744f9
Author: Tom Bentley 
Date:   2017-12-14T10:33:28Z

Fix to match KIP

commit 6096ef2a713e18f13de96d771244bc8e99742322
Author: Tom Bentley 
Date:   2017-12-14T12:54:15Z

WIP on algo improvment

commit 8c60b304283ed13e839b9ad3aee026f23dbaeb72
Author: Tom Bentley 
Date:   2017-12-14T13:21:54Z

Fixup

commit c0b6ac97e7bc8e168123ff6b6f467a035a73a02a
Author: Tom Bentley 
Date:   2017-12-15T09:26:19Z

Treat [1,2,3]->[1,3,2] as a no-op reassignment

Since only the first element in the list is distinguished.

commit b97521d81a3c7482f526d6574b3da34e1cfb9ff8
Author: Tom Bentley 
Date:   2017-12-15T09:28:02Z

Fixes

commit 52d61af3191d338222147395f3fe89b9f0015c80
Author: Tom Bentley 
Date:   2017-12-15T09:29:12Z

WIP: Change onPartitionReassignment() algo to cope with re-reassignment

commit 698cf3ef2b253347cddb0a49d5464fcd20781c05
Author: Tom Bentley 
Date:   2017-12-15T10:52:36Z

Refactor test

commit 117a1af116296795cfbd38ef85abd48d4b2643d0
Author: Tom Bentley 
Date:   2017-12-15T16:38:42Z

Improved algo




---

[GitHub] kafka pull request #4329: MINOR: stabilize flaky system tests

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/4329

MINOR: stabilize flaky system tests




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka hotfix-system-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4329.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4329


commit cb7ab621fde5e8a184edeb81715f86065a102a18
Author: Matthias J. Sax 
Date:   2017-12-15T18:08:02Z

MINOR: stabilize flaky system tests




---

Re: [DISCUSS] KIP-236 Interruptible Partition Reassignment

[Tom Bentley]

Just wanted to mention that I've started KIP-240, which builds on top of
this one to provide an AdminClient API for listing and describing
reassignments.

On 15 December 2017 at 14:34, Tom Bentley  wrote:

> > Should we seek to improve this algorithm in this KIP, or leave that as
> a later optimisation?
>
> I've updated the KIP with a proposed algorithm.
>
>
>
>
>
>
> On 14 December 2017 at 09:57, Tom Bentley  wrote:
>
>> Thanks Ted, now fixed.
>>
>> On 13 December 2017 at 18:38, Ted Yu  wrote:
>>
>>> Tom:
>>> bq. create a znode /admin/reassignments/$topic-$partition
>>>
>>> Looks like the tree structure above should be:
>>>
>>> /admin/reassignments/$topic/$partition
>>>
>>> bq. The controller removes /admin/reassignment/$topic/$partition
>>>
>>> Note the lack of 's' for reassignment. It would be good to make zookeeper
>>> paths consistent.
>>>
>>> Thanks
>>>
>>> On Wed, Dec 13, 2017 at 9:49 AM, Tom Bentley 
>>> wrote:
>>>
>>> > Hi Jun and Ted,
>>> >
>>> > Jun, you're right that needing one watcher per reassigned partition
>>> > presents a scalability problem, and using a separate notification path
>>> > solves that. I also agree that it makes sense to prevent users from
>>> using
>>> > both methods on the same reassignment.
>>> >
>>> > Ted, naming the reassignments like mytopic-42 was simpler while I was
>>> > proposing a watcher-per-reassignment (I'd have needed a child watcher
>>> on
>>> > /admin/reassignments and also on /admin/reassignments/mytopic). Using
>>> the
>>> > separate notification path means I don't need any watchers in the
>>> > /admin/reassignments subtree, so switching to
>>> /admin/reassignments/mytopic/
>>> > 42
>>> > would work, and avoid /admin/reassignments having a very large number
>>> of
>>> > child nodes. On the other hand it also means I have to create and
>>> delete
>>> > the topic nodes (e.g. /admin/reassignments/mytopic), which incurs the
>>> cost
>>> > of extra round trips to zookeeper. I suppose that since reassignment is
>>> > generally a slow process it makes little difference if we increase the
>>> > latency of the interactions with zookeeper.
>>> >
>>> > I have updated the KIP with these improvements, and a more detailed
>>> > description of exactly how we would manage these znodes.
>>> >
>>> > Reading the algorithm in KafkaController.onPartitionReassignment(), it
>>> > seems that it would be suboptimal for changing reassignments in-flight.
>>> > Consider an initial assignment of [1,2], reassigned to [2,3] and then
>>> > changed to [2,4]. Broker 3 will remain in the assigned replicas until
>>> > broker 4 is in sync, even though 3 wasn't actually one of the original
>>> > assigned replicas and is no longer a new assigned replica. I think this
>>> > also affects the case where the reassignment is cancelled
>>> > ([1,2]->[2,3]->[1,2]): We again have to wait for 3 to catch up, even
>>> though
>>> > its replica will then be deleted.
>>> >
>>> > Should we seek to improve this algorithm in this KIP, or leave that as
>>> a
>>> > later optimisation?
>>> >
>>> > Cheers,
>>> >
>>> > Tom
>>> >
>>> > On 11 December 2017 at 21:31, Jun Rao  wrote:
>>> >
>>> > > Another question is on the compatibility. Since now there are 2 ways
>>> of
>>> > > specifying a partition reassignment, one under
>>> /admin/reassign_partitions
>>> > > and the other under /admin/reassignments, we probably want to
>>> prevent the
>>> > > same topic being reassigned under both paths at the same time?
>>> > > Thanks,
>>> > >
>>> > > Jun
>>> > >
>>> > >
>>> > >
>>> > > On Fri, Dec 8, 2017 at 5:41 PM, Jun Rao  wrote:
>>> > >
>>> > > > Hi, Tom,
>>> > > >
>>> > > > Thanks for the KIP. It definitely addresses one of the pain points
>>> in
>>> > > > partition reassignment. Another issue that it also addresses is
>>> the ZK
>>> > > node
>>> > > > size limit when writing the reassignment JSON.
>>> > > >
>>> > > > My only concern is that the KIP needs to create one watcher per
>>> > > reassigned
>>> > > > partition. This could add overhead in ZK and complexity for
>>> debugging
>>> > > when
>>> > > > lots of partitions are being reassigned simultaneously. We could
>>> > > > potentially improve this by introducing a separate ZK path for
>>> change
>>> > > > notification as we do for configs. For example, every time we
>>> change
>>> > the
>>> > > > assignment for a set of partitions, we could further write a
>>> sequential
>>> > > > node /admin/reassignment_changes/[change_x]. That way, the
>>> controller
>>> > > > only needs to watch the change path. Once a change is triggered,
>>> the
>>> > > > controller can read everything under /admin/reassignments/.
>>> > > >
>>> > > > Jun
>>> > > >
>>> > > >
>>> > > > On Wed, Dec 6, 2017 at 1:19 PM, Tom Bentley >> >
>>> > > wrote:
>>> > > >
>>> > > >> Hi,
>>> > > >>
>>> > > >> This is still very new, but I wanted some quick 

[DISCUSS] KIP-240: AdminClient.listReassignments AdminClient.describeReassignments

[Tom Bentley]

Hi,

KIP-236 lays the foundations for AdminClient APIs to do with partition
reassignment. I'd now like to start discussing KIP-240, which adds APIs to
the AdminClient to list and describe the current reassignments.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-240%3A+AdminClient.listReassignments+AdminClient.describeReassignments

Aside: I have fairly developed ideas for the API for starting a
reassignment, but I intend to put that in a third KIP.

Cheers,

Tom

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

[dan]

i didn't think it would be appropriate to return `num.partitions` as a
Config for a Topic since that is not a valid config for that resource type.

yes, that is another option i have on the wiki in the Rejected Alternatives
section (though, its really just an alternative until something else passes
voting :). we still have the issue with return type here, NewTopic has the
info i want to return, but not necessarily the name i'd want... it could
make sense for a createTopics() to return a TopicDescription and we could
add the config/replication info to that class, but we'd then need to add
all this info for all describeTopics() calls. not sure of the
ramifications/overhead here.

dan

On Wed, Dec 13, 2017 at 10:10 AM, Colin McCabe  wrote:

> On Wed, Dec 13, 2017, at 10:00, dan wrote:
> > > Why not just return
> > > org.apache.kafka.clients.admin.Config like describeConfigs does?
> >
> > brokers have a `num.partitions` config that does not map to a valid
> > `Config` entry for a topic.
>
> Hi Dan,
>
> Sorry if I'm misunderstanding something, but why not map it to
> num.partitions?
>
> >
> > another added benefit to using `NewTopic` may be (future kip) having the
> > cluster return the actual replica mappings it would create (i have no
> > idea if this is actually possible)
>
> A better way of doing that would probably be extending
> CreateTopicsRequest so that it returns partition assignment information
> to the caller.  Then using validOnly = true to get this information.
>
> Actually, come to think of it, maybe we should be doing that for this
> KIP too.  Why not have CreateTopicsRequest return the config that was
> used, plus the partition assignment that was made?  We don't create
> topics that often, so the extra space on the wire should not be a
> concern.
>
> best,
> Colin
>
> >
> > dan
> >
> > On Wed, Dec 13, 2017 at 9:55 AM, Colin McCabe 
> wrote:
> >
> > > On Tue, Dec 12, 2017, at 19:02, Ewen Cheslack-Postava wrote:
> > > > re: API versions, I actually wasn't sure if we needed it or not. I'm
> fine
> > > > if people would prefer just bumping it, but I was actually curious
> if we
> > > > could get away without bumping it. I don't know the behavior of the
> > > > broker code paths for this well enough to know what types of errors
> those
> > > > non-null assertions get converted into.
> > >
> > > There's no advantage to trying to keep the API version number the same,
> > > though.  Since we have bidirectional client compatibility now, the
> > > clients and the server will just negotiate whatever version they need.
> > > New clients can still talk to older brokers that don't support this
> > > feature.
> > >
> > > If you don't bump the API version, the best case scenario is that you
> > > get a disconnect exception and the end-user is left confused about why.
> > > The worse-case scenario is that you crash the broker (but probably not,
> > > since you'd just get an NPE in serde, I think).  If you bump the
> version
> > > number, you can provide a proper UnsupportedVersionException when the
> > > feature is not supported.
> > >
> > > > For the return type, NewTopic seems reasonable and kind of intuitive
> --
> > > > basically a description of the NewTopic you would get. The only
> reason I
> > > > would be wary of reusing it is that what we don't want people doing
> is
> > > > taking that and passing it directly into AdminClient.createTopics
> since
> > > > we don't want them explicitly overriding all the defaults.
> > >
> > > Yeah.  Another thing is that NewTopic has a lot of stuff related to
> > > replication that doesn't seem relevant here.  For example, when
> creating
> > > NewTopic, you have the option of either setting replicationFactor, or
> > > setting up a specific replica assignment.  Why not just return
> > > org.apache.kafka.clients.admin.Config like describeConfigs does?
> > >
> > > best,
> > > Colin
> > >
> > > >
> > > > -Ewen
> > > >
> > > > On Tue, Dec 12, 2017 at 2:32 PM, dan  wrote:
> > > >
> > > > > Colin/Ewen,
> > > > >
> > > > > i will add changes to bump the API version.
> > > > >
> > > > > any preferences on the return type for the new method? tbh it seems
> > > like
> > > > > returning a NewTopic could make sense because the ConfigResource
> for a
> > > > > TOPIC type does not let me encode `numPartitions`
> > > > >
> > > > > thanks
> > > > > dan
> > > > >
> > > > > On Mon, Dec 11, 2017 at 7:22 PM, Colin McCabe 
> > > wrote:
> > > > >
> > > > > > Hi Dan,
> > > > > >
> > > > > > The KIP looks good overall.
> > > > > >
> > > > > > On Mon, Dec 11, 2017, at 18:28, Ewen Cheslack-Postava wrote:
> > > > > > > I think the key point is when the kafka admin and user creating
> > > topics
> > > > > > > differ. I think a more realistic example of Dan's point (2) is
> for
> > > > > > > retention. I know that realistically, admins aren't just going
> to
> > > > > > > randomly
> > > > > > > drop the broker 

Re: [VOTE] Allowing write access to GitHub repositories (aka GitBox)

[Ismael Juma]

Thanks to everyone who voted and contributed to the discussion.

The vote passes with 7 binding votes (Damian, Rajini, Jason, Gwen,
Guozhang, Sriram, Ismael) and 2 non-binding votes (Manikumar and Tom).

I will file a JIRA ticket in the Apache Infra project requesting the
migration to GitBox.

Ismael

On Thu, Dec 14, 2017 at 11:48 AM, Tom Bentley  wrote:

> +1
>
> On 12 December 2017 at 20:38, Sriram Subramanian  wrote:
>
> > +1
> >
> > On Tue, Dec 12, 2017 at 8:22 AM, Manikumar 
> > wrote:
> >
> > > +1
> > >
> > > On Tue, Dec 12, 2017 at 9:49 PM, Rajini Sivaram <
> rajinisiva...@gmail.com
> > >
> > > wrote:
> > >
> > > > +1
> > > >
> > > > Thanks, Ismael!
> > > >
> > > > On Tue, Dec 12, 2017 at 4:18 PM, Damian Guy 
> > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > On Tue, 12 Dec 2017 at 15:47 Ismael Juma 
> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > The Apache Infra team has started a new project earlier this year
> > > > called
> > > > > > GitBox that supports two-way synchronization between GitHub and
> > > > > > git-wip-us.apache.org and, most importantly, provides GitHub
> write
> > > > > access
> > > > > > to committers. GitBox is not generally available yet, but
> > individual
> > > > > > projects can ask to be migrated.
> > > > > >
> > > > > > I would like to start a vote on migrating kafka and kafka-site to
> > > > GitBox
> > > > > > and:
> > > > > >
> > > > > > 1. Providing GitHub write access to committers (this requires
> dual
> > > > factor
> > > > > > authentication)
> > > > > > 2. Allowing merges via the GitHub UI as well as the existing
> merge
> > > > script
> > > > > > 3. Enabling protected branches for trunk and release branches so
> > that
> > > > > > merges via the GitHub UI can only be done if the tests pass and
> the
> > > PR
> > > > > has
> > > > > > been approved by a committer
> > > > > > 4. Only allowing the "squash and merge" strategy for GitHub UI
> > merges
> > > > > > 5. Updating the merge script so that the GitHub git repo is the
> > > target
> > > > of
> > > > > > the merge
> > > > > > 6. Disallowing force pushes to trunk and release branches
> > > > > >
> > > > > > The discussion thread talks about some of the pros and cons
> (mostly
> > > > pros)
> > > > > > of this change:
> > > > > >
> > > > > >
> > > > > > https://lists.apache.org/thread.html/
> > 7031168e7026222169c66fed29f520
> > > > > 0fc4b561df28c242ccf706f326@%3Cdev.kafka.apache.org%3E
> > > > > >
> > > > > > The vote will run for 72 hours.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-236 Interruptible Partition Reassignment

[Tom Bentley]

> Should we seek to improve this algorithm in this KIP, or leave that as a
later optimisation?

I've updated the KIP with a proposed algorithm.






On 14 December 2017 at 09:57, Tom Bentley  wrote:

> Thanks Ted, now fixed.
>
> On 13 December 2017 at 18:38, Ted Yu  wrote:
>
>> Tom:
>> bq. create a znode /admin/reassignments/$topic-$partition
>>
>> Looks like the tree structure above should be:
>>
>> /admin/reassignments/$topic/$partition
>>
>> bq. The controller removes /admin/reassignment/$topic/$partition
>>
>> Note the lack of 's' for reassignment. It would be good to make zookeeper
>> paths consistent.
>>
>> Thanks
>>
>> On Wed, Dec 13, 2017 at 9:49 AM, Tom Bentley 
>> wrote:
>>
>> > Hi Jun and Ted,
>> >
>> > Jun, you're right that needing one watcher per reassigned partition
>> > presents a scalability problem, and using a separate notification path
>> > solves that. I also agree that it makes sense to prevent users from
>> using
>> > both methods on the same reassignment.
>> >
>> > Ted, naming the reassignments like mytopic-42 was simpler while I was
>> > proposing a watcher-per-reassignment (I'd have needed a child watcher on
>> > /admin/reassignments and also on /admin/reassignments/mytopic). Using
>> the
>> > separate notification path means I don't need any watchers in the
>> > /admin/reassignments subtree, so switching to
>> /admin/reassignments/mytopic/
>> > 42
>> > would work, and avoid /admin/reassignments having a very large number of
>> > child nodes. On the other hand it also means I have to create and delete
>> > the topic nodes (e.g. /admin/reassignments/mytopic), which incurs the
>> cost
>> > of extra round trips to zookeeper. I suppose that since reassignment is
>> > generally a slow process it makes little difference if we increase the
>> > latency of the interactions with zookeeper.
>> >
>> > I have updated the KIP with these improvements, and a more detailed
>> > description of exactly how we would manage these znodes.
>> >
>> > Reading the algorithm in KafkaController.onPartitionReassignment(), it
>> > seems that it would be suboptimal for changing reassignments in-flight.
>> > Consider an initial assignment of [1,2], reassigned to [2,3] and then
>> > changed to [2,4]. Broker 3 will remain in the assigned replicas until
>> > broker 4 is in sync, even though 3 wasn't actually one of the original
>> > assigned replicas and is no longer a new assigned replica. I think this
>> > also affects the case where the reassignment is cancelled
>> > ([1,2]->[2,3]->[1,2]): We again have to wait for 3 to catch up, even
>> though
>> > its replica will then be deleted.
>> >
>> > Should we seek to improve this algorithm in this KIP, or leave that as a
>> > later optimisation?
>> >
>> > Cheers,
>> >
>> > Tom
>> >
>> > On 11 December 2017 at 21:31, Jun Rao  wrote:
>> >
>> > > Another question is on the compatibility. Since now there are 2 ways
>> of
>> > > specifying a partition reassignment, one under
>> /admin/reassign_partitions
>> > > and the other under /admin/reassignments, we probably want to prevent
>> the
>> > > same topic being reassigned under both paths at the same time?
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > >
>> > >
>> > > On Fri, Dec 8, 2017 at 5:41 PM, Jun Rao  wrote:
>> > >
>> > > > Hi, Tom,
>> > > >
>> > > > Thanks for the KIP. It definitely addresses one of the pain points
>> in
>> > > > partition reassignment. Another issue that it also addresses is the
>> ZK
>> > > node
>> > > > size limit when writing the reassignment JSON.
>> > > >
>> > > > My only concern is that the KIP needs to create one watcher per
>> > > reassigned
>> > > > partition. This could add overhead in ZK and complexity for
>> debugging
>> > > when
>> > > > lots of partitions are being reassigned simultaneously. We could
>> > > > potentially improve this by introducing a separate ZK path for
>> change
>> > > > notification as we do for configs. For example, every time we change
>> > the
>> > > > assignment for a set of partitions, we could further write a
>> sequential
>> > > > node /admin/reassignment_changes/[change_x]. That way, the
>> controller
>> > > > only needs to watch the change path. Once a change is triggered, the
>> > > > controller can read everything under /admin/reassignments/.
>> > > >
>> > > > Jun
>> > > >
>> > > >
>> > > > On Wed, Dec 6, 2017 at 1:19 PM, Tom Bentley 
>> > > wrote:
>> > > >
>> > > >> Hi,
>> > > >>
>> > > >> This is still very new, but I wanted some quick feedback on a
>> > > preliminary
>> > > >> KIP which could, I think, help with providing an AdminClient API
>> for
>> > > >> partition reassignment.
>> > > >>
>> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-236%
>> > > >> 3A+Interruptible+Partition+Reassignment
>> > > >>
>> > > >> I wasn't sure whether to start fleshing out a whole AdminClient
>> API in
>> > > >> this
>> > > >> 

Build failed in Jenkins: kafka-trunk-jdk8 #2278

[Apache Jenkins Server]

See 


Changes:

[damian.guy] KAFKA-6086: Provide for custom error handling when Kafka Streams 
fails

--
[...truncated 404.40 KB...]
kafka.utils.CoreUtilsTest > testAtomicGetOrUpdate PASSED

kafka.utils.CoreUtilsTest > testUrlSafeBase64EncodeUUID STARTED

kafka.utils.CoreUtilsTest > testUrlSafeBase64EncodeUUID PASSED

kafka.utils.CoreUtilsTest > testCsvMap STARTED

kafka.utils.CoreUtilsTest > testCsvMap PASSED

kafka.utils.CoreUtilsTest > testInLock STARTED

kafka.utils.CoreUtilsTest > testInLock PASSED

kafka.utils.CoreUtilsTest > testTryAll STARTED

kafka.utils.CoreUtilsTest > testTryAll PASSED

kafka.utils.CoreUtilsTest > testSwallow STARTED

kafka.utils.CoreUtilsTest > testSwallow PASSED

kafka.utils.IteratorTemplateTest > testIterator STARTED

kafka.utils.IteratorTemplateTest > testIterator PASSED

kafka.utils.json.JsonValueTest > testJsonObjectIterator STARTED

kafka.utils.json.JsonValueTest > testJsonObjectIterator PASSED

kafka.utils.json.JsonValueTest > testDecodeLong STARTED

kafka.utils.json.JsonValueTest > testDecodeLong PASSED

kafka.utils.json.JsonValueTest > testAsJsonObject STARTED

kafka.utils.json.JsonValueTest > testAsJsonObject PASSED

kafka.utils.json.JsonValueTest > testDecodeDouble STARTED

kafka.utils.json.JsonValueTest > testDecodeDouble PASSED

kafka.utils.json.JsonValueTest > testDecodeOption STARTED

kafka.utils.json.JsonValueTest > testDecodeOption PASSED

kafka.utils.json.JsonValueTest > testDecodeString STARTED

kafka.utils.json.JsonValueTest > testDecodeString PASSED

kafka.utils.json.JsonValueTest > testJsonValueToString STARTED

kafka.utils.json.JsonValueTest > testJsonValueToString PASSED

kafka.utils.json.JsonValueTest > testAsJsonObjectOption STARTED

kafka.utils.json.JsonValueTest > testAsJsonObjectOption PASSED

kafka.utils.json.JsonValueTest > testAsJsonArrayOption STARTED

kafka.utils.json.JsonValueTest > testAsJsonArrayOption PASSED

kafka.utils.json.JsonValueTest > testAsJsonArray STARTED

kafka.utils.json.JsonValueTest > testAsJsonArray PASSED

kafka.utils.json.JsonValueTest > testJsonValueHashCode STARTED

kafka.utils.json.JsonValueTest > testJsonValueHashCode PASSED

kafka.utils.json.JsonValueTest > testDecodeInt STARTED

kafka.utils.json.JsonValueTest > testDecodeInt PASSED

kafka.utils.json.JsonValueTest > testDecodeMap STARTED

kafka.utils.json.JsonValueTest > testDecodeMap PASSED

kafka.utils.json.JsonValueTest > testDecodeSeq STARTED

kafka.utils.json.JsonValueTest > testDecodeSeq PASSED

kafka.utils.json.JsonValueTest > testJsonObjectGet STARTED

kafka.utils.json.JsonValueTest > testJsonObjectGet PASSED

kafka.utils.json.JsonValueTest > testJsonValueEquals STARTED

kafka.utils.json.JsonValueTest > testJsonValueEquals PASSED

kafka.utils.json.JsonValueTest > testJsonArrayIterator STARTED

kafka.utils.json.JsonValueTest > testJsonArrayIterator PASSED

kafka.utils.json.JsonValueTest > testJsonObjectApply STARTED

kafka.utils.json.JsonValueTest > testJsonObjectApply PASSED

kafka.utils.json.JsonValueTest > testDecodeBoolean STARTED

kafka.utils.json.JsonValueTest > testDecodeBoolean PASSED

kafka.producer.AsyncProducerTest > testFailedSendRetryLogic STARTED

kafka.producer.AsyncProducerTest > testFailedSendRetryLogic PASSED

kafka.producer.AsyncProducerTest > testQueueTimeExpired STARTED

kafka.producer.AsyncProducerTest > testQueueTimeExpired FAILED
java.lang.AssertionError: 
  Expectation failure on verify:

DefaultEventHandler.handle(ArrayBuffer(KeyedMessage(topic1,null,null,msg0), 
KeyedMessage(topic1,null,null,msg1))): expected: 1, actual: 1
at org.easymock.internal.MocksControl.verify(MocksControl.java:277)
at org.easymock.EasyMock.verify(EasyMock.java:2041)
at 
kafka.producer.AsyncProducerTest.testQueueTimeExpired(AsyncProducerTest.scala:157)

kafka.producer.AsyncProducerTest > testPartitionAndCollateEvents STARTED

kafka.producer.AsyncProducerTest > testPartitionAndCollateEvents PASSED

kafka.producer.AsyncProducerTest > testBatchSize STARTED

kafka.producer.AsyncProducerTest > testBatchSize PASSED

kafka.producer.AsyncProducerTest > testSerializeEvents STARTED

kafka.producer.AsyncProducerTest > testSerializeEvents PASSED

kafka.producer.AsyncProducerTest > testProducerQueueSize STARTED

kafka.producer.AsyncProducerTest > testProducerQueueSize PASSED

kafka.producer.AsyncProducerTest > testRandomPartitioner STARTED

kafka.producer.AsyncProducerTest > testRandomPartitioner PASSED

kafka.producer.AsyncProducerTest > testInvalidConfiguration STARTED

kafka.producer.AsyncProducerTest > testInvalidConfiguration PASSED

kafka.producer.AsyncProducerTest > testInvalidPartition STARTED

kafka.producer.AsyncProducerTest > testInvalidPartition PASSED

kafka.producer.AsyncProducerTest > testNoBroker STARTED

kafka.producer.AsyncProducerTest > testNoBroker PASSED

kafka.producer.AsyncProducerTest > 

Build failed in Jenkins: kafka-trunk-jdk7 #3040

[Apache Jenkins Server]

See 


Changes:

[damian.guy] KAFKA-6086: Provide for custom error handling when Kafka Streams 
fails

--
[...truncated 404.02 KB...]
kafka.log.BrokerCompressionTest > testBrokerSideCompression[17] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[17] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[18] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[18] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[19] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[19] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[0] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[0] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[0] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[0] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[0] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[0] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[1] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[1] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[1] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[1] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[1] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[1] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[2] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[2] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[2] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[2] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[2] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[2] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[3] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleansCombinedCompactAndDeleteTopic[3] PASSED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[3] STARTED

kafka.log.LogCleanerIntegrationTest > 
testCleaningNestedMessagesWithMultipleVersions[3] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[3] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[3] PASSED

kafka.log.ProducerStateManagerTest > testCoordinatorFencing STARTED

kafka.log.ProducerStateManagerTest > testCoordinatorFencing PASSED

kafka.log.ProducerStateManagerTest > testTruncate STARTED

kafka.log.ProducerStateManagerTest > testTruncate PASSED

kafka.log.ProducerStateManagerTest > testLoadFromTruncatedSnapshotFile STARTED

kafka.log.ProducerStateManagerTest > testLoadFromTruncatedSnapshotFile PASSED

kafka.log.ProducerStateManagerTest > testRemoveExpiredPidsOnReload STARTED

kafka.log.ProducerStateManagerTest > testRemoveExpiredPidsOnReload PASSED

kafka.log.ProducerStateManagerTest > 
testOutOfSequenceAfterControlRecordEpochBump STARTED

kafka.log.ProducerStateManagerTest > 
testOutOfSequenceAfterControlRecordEpochBump PASSED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffsetAfterTruncation 
STARTED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffsetAfterTruncation 
PASSED

kafka.log.ProducerStateManagerTest > testTakeSnapshot STARTED

kafka.log.ProducerStateManagerTest > testTakeSnapshot PASSED

kafka.log.ProducerStateManagerTest > testDeleteSnapshotsBefore STARTED

kafka.log.ProducerStateManagerTest > testDeleteSnapshotsBefore PASSED

kafka.log.ProducerStateManagerTest > 
testNonMatchingTxnFirstOffsetMetadataNotCached STARTED

kafka.log.ProducerStateManagerTest > 
testNonMatchingTxnFirstOffsetMetadataNotCached PASSED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffsetAfterEviction 
STARTED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffsetAfterEviction PASSED

kafka.log.ProducerStateManagerTest > testNoValidationOnFirstEntryWhenLoadingLog 
STARTED

kafka.log.ProducerStateManagerTest > 

[GitHub] kafka pull request #4165: KAFKA-6086: Provide for custom error handling when...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4165


---

Re: [VOTE] KIP-225 - Use tags for consumer “records.lag” metrics

[charly molter]

Thank you very much for the votes.

The vote is now closed with 3 binding +1 (Becket Qin, Jun Rao and Jason
Gustafson).

On Mon, Dec 11, 2017 at 11:21 PM, Jason Gustafson 
wrote:

> +1. Thanks for the KIP.
>
> On Mon, Dec 11, 2017 at 1:54 AM, charly molter 
> wrote:
>
> > Hi,
> > The KIP has been updated. As it has change should I restart the vote?
> >
> > In any case I'm still missing one binding vote if anyone wants to help.
> > Thanks!
> >
> > On Wed, Dec 6, 2017 at 6:13 PM, charly molter 
> > wrote:
> >
> > > Sounds good I'll update the KIP
> > >
> > > On Wed, Dec 6, 2017 at 6:04 PM, Becket Qin 
> wrote:
> > >
> > >> Hi Charly,
> > >>
> > >> Personally I prefer emitting both and deprecate old one. This does not
> > >> block on the 2.0 release and we don't need to worry about more users
> > >> picking up the old metric in 1.1 release.
> > >>
> > >> Thanks,
> > >>
> > >> Jiangjie (Becket) Qin
> > >>
> > >> On Tue, Dec 5, 2017 at 4:08 AM, charly molter <
> charly.mol...@gmail.com>
> > >> wrote:
> > >>
> > >> > Thanks Jun and Becket!
> > >> >
> > >> > I think your point about 1.0 vs 2.0 makes sense I can update the KIP
> > to
> > >> > reflect this.
> > >> >
> > >> > What's the process for 2.0 contributions as I can see that trunk is
> > 1.1
> > >> and
> > >> > no 2.x branch?
> > >> >
> > >> > Here's what I can do:
> > >> > - Not write the code change until trunk moves to 2.0.
> > >> > - Write the change but leave the PR open until we start working on
> > 2.0.
> > >> > - Stall this KIP until 2.0 development starts (IIRC it's pretty
> soon).
> > >> > - Do it in a backward compatible way (publish both sets of metrics)
> > and
> > >> > open a Jira tagged on 2.0 to remove the old metrics.
> > >> >
> > >> > Let me know what's the right way to go.
> > >> >
> > >> > Thanks!
> > >> >
> > >> >
> > >> > On Tue, Dec 5, 2017 at 12:45 AM, Becket Qin 
> > >> wrote:
> > >> >
> > >> > > Thanks for the KIP, Charly.
> > >> > >
> > >> > > +1. The proposal looks good to me. I agree with Jun that it is
> > better
> > >> to
> > >> > > make the metrics consistent with other metrics. That being said,
> > >> arguably
> > >> > > this is a backwards incompatible change. Since we are at 1.0,
> > >> backwards
> > >> > > incompatible changes are supposed to be in 2.0. Not sure if that
> is
> > >> the
> > >> > > plan or not.
> > >> > >
> > >> > > Thanks,
> > >> > >
> > >> > > Jiangjie (Becket) Qin
> > >> > >
> > >> > > On Mon, Dec 4, 2017 at 4:20 PM, Jun Rao  wrote:
> > >> > >
> > >> > > > Hi, Jiangjie,
> > >> > > >
> > >> > > > Since you proposed the original KIP-92, do you want to see if
> this
> > >> KIP
> > >> > > > makes sense?
> > >> > > >
> > >> > > > Thanks,
> > >> > > >
> > >> > > > Jun
> > >> > > >
> > >> > > > On Wed, Nov 22, 2017 at 2:48 AM, charly molter <
> > >> > charly.mol...@gmail.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Hi,
> > >> > > > >
> > >> > > > > I would like to start the voting thread for KIP-225.
> > >> > > > > This KIP proposes to correct some lag metrics emitted by the
> > >> > consumer.
> > >> > > > >
> > >> > > > > The KIP wiki is here:
> > >> > > > > https://cwiki.apache.org/confluence/x/uaBzB
> > >> > > > >
> > >> > > > > The discussion thread is here:
> > >> > > > > http://search-hadoop.com/m/Kafka/uyzND1F33uL19AYx/threaded
> > >> > > > >
> > >> > > > > Also could someone assign me to this Jira: KAFKA-5890
> > >> > > > > 
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > --
> > >> > > > > Charly Molter
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Charly Molter
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Charly Molter
> > >
> >
> >
> >
> > --
> > Charly Molter
> >
>



-- 
Charly Molter

[GitHub] kafka pull request #4328: KAFKA-6371 Fix DelayedFetch toString

[mayt]

GitHub user mayt opened a pull request:

https://github.com/apache/kafka/pull/4328

KAFKA-6371 Fix DelayedFetch toString




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mayt/kafka KAFKA-6371

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4328.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4328






---

[jira] [Created] (KAFKA-6371) FetchMetadata creates unneeded Strings on instantiation

[Maytee Chinavanichkit (JIRA)]

Maytee Chinavanichkit created KAFKA-6371:


 Summary: FetchMetadata creates unneeded Strings on instantiation
 Key: KAFKA-6371
 URL: https://issues.apache.org/jira/browse/KAFKA-6371
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.10.2.1, 0.10.1.1, 0.10.0.1
Reporter: Maytee Chinavanichkit
Assignee: Maytee Chinavanichkit
Priority: Minor
 Fix For: 0.10.0.2, 0.10.1.2, 0.10.2.2


My colleague and I were taking a heap dump to investigate the memory usage of a 
broker. From the dump, we saw a number of object strings with the message 
{{onlyCommitted: }} and {{partitionStatus: ]}}. Upon 
investigation, these objects were being instantiated when the {{FetchMetadata}} 
object is constructed. The toString method here is malformed and the last two 
lines are executed as a block instead of being concatenated. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #4327: KAFKA-6370: KafkaMetricsGroup.toScope should filte...

[huxihx]

GitHub user huxihx opened a pull request:

https://github.com/apache/kafka/pull/4327

KAFKA-6370: KafkaMetricsGroup.toScope should filter out tags null value

KafkaMetricsGroup.toScope should filter out tags with value of `null` to 
avoid NullPointerException thrown.

*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*

*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*

### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation 
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/huxihx/kafka KAFKA-6370

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4327.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4327


commit 9eec15c5f231cb6c5ec17cc62783031775d69b74
Author: huxihx 
Date:   2017-12-15T08:05:52Z

KAFKA-6370: KafkaMetricsGroup.toScope should filter out tags with value of 
`null` to avoid NullPointerException thrown.




---