[jira] [Commented] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)
[ https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049874#comment-16049874 ] ASF GitHub Bot commented on KAFKA-5275: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3339 > Review and potentially tweak AdminClient API for the initial release (KIP-117) > -- > > Key: KAFKA-5275 > URL: https://issues.apache.org/jira/browse/KAFKA-5275 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.11.0.0 > > > Once all the pieces are in, we should take a pass and ensure that the APIs > work well together and that they are consistent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)
[ https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049301#comment-16049301 ] ASF GitHub Bot commented on KAFKA-5275: --- GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/3339 KAFKA-5275: AdminClient API consistency You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijuma/kafka kafka-5275-admin-client-api-consistency Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3339.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3339 commit 515037d6c8d37c4d088fd1d15b2275a609ac7e26 Author: Ismael JumaDate: 2017-06-13T13:56:05Z Publish javadoc for common.annotation package It includes InterfaceStability annotation. commit 51264deb6a49221d9d524896f91fd70af2439887 Author: Ismael Juma Date: 2017-06-13T14:10:44Z Clarify InterfaceStability commit c5d77c5cb28bf5db2932c1862e0ff44c74513158 Author: Ismael Juma Date: 2017-06-13T14:27:46Z Various javadoc improvements to API classes in clients.admin commit 57d83d031fee20ac4e988c75deb6870c0be8ed41 Author: Ismael Juma Date: 2017-06-13T23:40:40Z Revert assert change commit d5a1bb479f968ee0da86542446befc7e4ea77019 Author: Ismael Juma Date: 2017-06-14T00:06:20Z Add javadoc to some classes in common commit a5bb109c46f90f0a9aa7312b725bc8326feb17e2 Author: Ismael Juma Date: 2017-06-14T00:31:51Z More javadoc for common classes commit baffc69ef6cbaf4d5b8478bef6a7cfc6fa639fed Author: Ismael Juma Date: 2017-06-14T00:33:26Z Address review feedback commit ac064599f1708ae8e7a1194922a138af92999d37 Author: Ismael Juma Date: 2017-06-14T00:56:41Z Document TopicPartitionInfo commit 72a0010ad5a0ddd2b18e0c757c70a923e4876d8c Author: Ismael Juma Date: 2017-06-14T12:34:40Z Document broker requirement for AdminClient methods commit 1a65c1005c65d03196e3db2cee2387389940421c Author: Ismael Juma Date: 2017-06-14T14:38:06Z Add InterfaceStability to more classes commit bdf4d09b3139d13c215f5ec7faea30654aa6db34 Author: Ismael Juma Date: 2017-06-14T13:11:04Z Use List instead of NavigableMap for TopicDescription.partitions commit 5d5bd02bfc7ef657514e3ffd3ccffab18957b486 Author: Ismael Juma Date: 2017-06-14T13:11:37Z Lists exposed by TopicPartitionInfo should be unmodifiable commit f41a6651728eecc0f16bbe167aaf747dc29ed8a1 Author: Ismael Juma Date: 2017-06-14T13:17:15Z Rename TopicListing to TopicListItem Listing doesn't seem to be the right term for what it represents. commit 5bfb60756deee38394fb00efb7dd2d5871edae95 Author: Ismael Juma Date: 2017-06-14T13:18:31Z Rename NewTopic.partitions to NewTopic.numPartitions commit 89b88deafa9cadeef2c4f35f75bc0e0331cb2456 Author: Ismael Juma Date: 2017-06-14T13:24:42Z Replace `description` usage in `ListTopicsResult` `ListTopicsResult` doesn't return `TopicDescription` commit e75985e0a1b6fe40d3309d36bf17e84e856edbd8 Author: Ismael Juma Date: 2017-06-14T13:36:24Z Rename `results()` to `value()` commit 93fa6cafc908958ab81f416a76e4d41c1dae28ed Author: Ismael Juma Date: 2017-06-14T13:43:36Z Don't use JVM level asserts as they are not enabled by default commit 50db3ccac1d168cc12109d99a5a3c40260c5b781 Author: Ismael Juma Date: 2017-06-14T14:07:32Z Make retries configurable commit 8752681528573e937b7ab5c1c7c9548fa21cbf29 Author: Ismael Juma Date: 2017-06-14T15:20:41Z Consistent usage of prefix for boolean accessors The other option is to remove all of the existing prefixes. commit c55861a3461931d22e340d6b7af8eb4e17c7b51c Author: Ismael Juma Date: 2017-06-14T15:36:11Z Use `null` for unknown controller or leader > Review and potentially tweak AdminClient API for the initial release (KIP-117) > -- > > Key: KAFKA-5275 > URL: https://issues.apache.org/jira/browse/KAFKA-5275 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.11.0.0 > > > Once all the pieces are in, we should take a pass and ensure that the APIs > work well together and that they are consistent. -- This message was sent by Atlassian
[jira] [Commented] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)
[ https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037820#comment-16037820 ] Xavier Léauté commented on KAFKA-5275: -- Now that describe topic doesn't auto-create the topic anymore it's easier to implement a "create topic if not exists" functionality that doesn't require listing all topics. However given the nature of the async API, it still feels cumbersome to write, e.g. {code} try { adminClient.describeTopics(Collections.singleton(publishTopic)).all().get(); log.debug("Metrics reporter topic {} already exists", publishTopic); } catch (ExecutionException e) { if (!(e.getCause() instanceof UnknownTopicOrPartitionException)) { // something bad happened throw e; } // create logic here } {code} > Review and potentially tweak AdminClient API for the initial release (KIP-117) > -- > > Key: KAFKA-5275 > URL: https://issues.apache.org/jira/browse/KAFKA-5275 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.11.0.0 > > > Once all the pieces are in, we should take a pass and ensure that the APIs > work well together and that they are consistent. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)
[ https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020137#comment-16020137 ] Colin P. McCabe commented on KAFKA-5275: I can imagine clients handling some types of exceptions. For example, an {{InvalidConfigurationException}} includes the name of the unknown configuration key, so you could strip that configuration key off and retry. Maybe you don't care about {{TopicExistsException}}, so you log that and ignore it. If you get a subclass of {{RetriableException}}, you can retry at the application level (these are nearly always fallout from network issues which could be transitory.) If you get {{AuthenticationException}}, you might want to display a nicer message to the user. Of course, {{UnknownServerException}} really is unhandleable, but it also shouldn't happen that often. > Review and potentially tweak AdminClient API for the initial release (KIP-117) > -- > > Key: KAFKA-5275 > URL: https://issues.apache.org/jira/browse/KAFKA-5275 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.11.0.0 > > > Once all the pieces are in, we should take a pass and ensure that the APIs > work well together and that they are consistent. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)
[ https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020058#comment-16020058 ] Randall Hauch commented on KAFKA-5275: -- [~cmccabe] wrote: {quote} Hmm. Methods like these, can't handle the case where there is an error other than TopicExistsException. They have to throw an exception, and then you get no information about any topic other than the one that had the exception. Can't you just iterate through the map and do per-topic error handling? It just seems like an easier and cleaner solution, unless there's something I'm missing. {quote} Yes, that's true that consumers don't really know how to handle any kind of exception, so I would assume that they just log the exceptions and continue or fail by rethrowing them. However, you're right that we wouldn't want to lose any of the exceptions, so if we don't provide an exception handler or can't aggregate them somehow into a single exception then we probably shouldn't offer the capability to create topics this way. > Review and potentially tweak AdminClient API for the initial release (KIP-117) > -- > > Key: KAFKA-5275 > URL: https://issues.apache.org/jira/browse/KAFKA-5275 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.11.0.0 > > > Once all the pieces are in, we should take a pass and ensure that the APIs > work well together and that they are consistent. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)
[ https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16020039#comment-16020039 ] Colin P. McCabe commented on KAFKA-5275: bq. [~xvl] wrote: To echo Randall's point about having a topic config builder, I completely agree. However, we should also define constants for every topic configuration option that exists. Currently the only way to get those is to pull in the kafka-server jar and access things like LogConfig.SegmentMsProp() directly That's a very good point. It would be much more user-friendly to provide constants. We need this anyway for things like {{alterConfigs}} and {{describeConfigs}}. Maybe we could do this by putting all the constants into {{AdminClientConf}}. Since the server jars depend on the client jars, but not vice versa, we would then have to change {{KafkaConfig.scala}} so that it defined all these constants in terms of the {{AdminClientConf}} versions. That way, everything would stay in sync. bq. [~rhauch] wrote: Kafka Connect does log which topics were created and which were found. That'd be trivial if CreateTopicResults had methods to return the names of the newly-created and existing topics – perhaps something like: createdTopicNames, existingTopicNames Hmm. Methods like these, can't handle the case where there is an error other than {{TopicExistsException}}. They have to throw an exception, and then you get no information about any topic other than the one that had the exception. Can't you just iterate through the map and do per-topic error handling? It just seems like an easier and cleaner solution, unless there's something I'm missing. > Review and potentially tweak AdminClient API for the initial release (KIP-117) > -- > > Key: KAFKA-5275 > URL: https://issues.apache.org/jira/browse/KAFKA-5275 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.11.0.0 > > > Once all the pieces are in, we should take a pass and ensure that the APIs > work well together and that they are consistent. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)
[ https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017701#comment-16017701 ] Randall Hauch commented on KAFKA-5275: -- [~colinmccabe] wrote: {quote} We could add an option to {{CreateTopicsOptions}} that suppresses {{TopicExistsException}} in the results. I guess the question is, does your code want to at least log something when a topic already exists rather than getting created, since the configuration options will potentially be different... {quote} Kafka Connect does log which topics were created and which were found. That'd be trivial if {{CreateTopicResults}} had methods to return the names of the newly-created and existing topics -- perhaps something like: {code:java} KafkaFuturecreatedTopicNames() {...} KafkaFuture existingTopicNames() {...} {code} [~xvrl] wrote: {quote} However, we should also define constants for every topic configuration option that exists. {quote} Yes, and ideally the NewTopic builder might even have some convenience methods that saved having to use some of them. > Review and potentially tweak AdminClient API for the initial release (KIP-117) > -- > > Key: KAFKA-5275 > URL: https://issues.apache.org/jira/browse/KAFKA-5275 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.11.0.0 > > > Once all the pieces are in, we should take a pass and ensure that the APIs > work well together and that they are consistent. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)
[ https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017659#comment-16017659 ] Xavier Léauté commented on KAFKA-5275: -- To echo Randall's point about having a topic config builder, I completely agree. However, we should also define constants for every topic configuration option that exists. Currently the only way to get those is to pull in the kafka-server jar and access things like {{LogConfig.SegmentMsProp()}} directly I believe it would be useful to keep the ability to set arbitrary string configuration to be forward-compatible and not require someone to upgrade their client just to set a new topic option. > Review and potentially tweak AdminClient API for the initial release (KIP-117) > -- > > Key: KAFKA-5275 > URL: https://issues.apache.org/jira/browse/KAFKA-5275 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.11.0.0 > > > Once all the pieces are in, we should take a pass and ensure that the APIs > work well together and that they are consistent. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)
[ https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017646#comment-16017646 ] Xavier Léauté commented on KAFKA-5275: -- Speaking of {{TopicExistsException}} it would also be nice to document the possible failure modes of the various api methods, so one doesn't have to find out by trial and error. It's also not immediately clear which recoverable errors the admin client retries automatically for the user vs. which ones need to be handled by the user. > Review and potentially tweak AdminClient API for the initial release (KIP-117) > -- > > Key: KAFKA-5275 > URL: https://issues.apache.org/jira/browse/KAFKA-5275 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.11.0.0 > > > Once all the pieces are in, we should take a pass and ensure that the APIs > work well together and that they are consistent. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)
[ https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017629#comment-16017629 ] Colin P. McCabe commented on KAFKA-5275: bq. It'd be great to simplify the creation of the NewTopic objects. As part of the PR for KIP-154 / KAFKA-4667 I added a builder for NewTopic that has methods to set replication factor, num partitions, the cleanup policy (currently compacted only), min ISRs, etc. It's currently an implementation detail of Kafka Connect, but it'd be great if that could move down to the AdminClient. Yeah, I think it makes sense to consider adding this to `AdminClient`. [~xvrl] also proposed adding a comment to the {{AdminClient}} JavaDoc discussing the existence of the {{AdminClientConfig}} class, which might not be clear for first-time users. > Review and potentially tweak AdminClient API for the initial release (KIP-117) > -- > > Key: KAFKA-5275 > URL: https://issues.apache.org/jira/browse/KAFKA-5275 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.11.0.0 > > > Once all the pieces are in, we should take a pass and ensure that the APIs > work well together and that they are consistent. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)
[ https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017626#comment-16017626 ] Colin P. McCabe commented on KAFKA-5275: bq. [~rhauch]: Also, a common use case for creating topics is going to be "create new topic(s) if they don't already exist". The current AdminClient can do this, but it requires a fair amount of error handling, especially when trying to create multiple topics at once. We could add an option to {{CreateTopicsOptions}} that suppresses {{TopicExistsException}} in the results. I guess the question is, does your code want to at least log something when a topic already exists rather than getting created, since the configuration options will potentially be different... bq. it would be nice to have a default describeTopics() that doesn't need a list of topics and just returns the description for all topics, right now i have to do adminClient.describeTopics(a.listTopics().names().get()).all().get(); :/ The intention here was to encourage people to fetch a few topics at a time, since we don't currently have very scalable ways of fetching all topics. KIP-142 will help here. Of course, sometimes you really do need to fetch every single topic in the cluster. We probably should have some nicer way of doing this. Perhaps it could do batching in the AdminClient. > Review and potentially tweak AdminClient API for the initial release (KIP-117) > -- > > Key: KAFKA-5275 > URL: https://issues.apache.org/jira/browse/KAFKA-5275 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.11.0.0 > > > Once all the pieces are in, we should take a pass and ensure that the APIs > work well together and that they are consistent. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)
[ https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16016718#comment-16016718 ] dan norwood commented on KAFKA-5275: it would be nice to have a default {{describeTopics()}} that doesn't need a list of topics and just returns the description for all topics, right now i have to do {{adminClient.describeTopics(a.listTopics().names().get()).all().get();}} :/ > Review and potentially tweak AdminClient API for the initial release (KIP-117) > -- > > Key: KAFKA-5275 > URL: https://issues.apache.org/jira/browse/KAFKA-5275 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.11.0.0 > > > Once all the pieces are in, we should take a pass and ensure that the APIs > work well together and that they are consistent. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)
[ https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16016513#comment-16016513 ] Randall Hauch commented on KAFKA-5275: -- Also, a common use case for creating topics is going to be "create new topic(s) if they don't already exist". The current AdminClient can do this, but it requires a fair amount of error handling, especially when trying to create multiple topics at once. Would be great if this was easier, even if it's just an additional method on {{CreateTopicResults}} that perhaps gets the names of the topics that were created as part of the request. Again, see Kafka Connect's {{TopicAdmin}} code for an example that does this. > Review and potentially tweak AdminClient API for the initial release (KIP-117) > -- > > Key: KAFKA-5275 > URL: https://issues.apache.org/jira/browse/KAFKA-5275 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.11.0.0 > > > Once all the pieces are in, we should take a pass and ensure that the APIs > work well together and that they are consistent. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)
[ https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16016495#comment-16016495 ] Randall Hauch commented on KAFKA-5275: -- It'd be great to simplify the creation of the {{NewTopic}} objects. As part of the [PR|https://github.com/apache/kafka/pull/2984] for [KIP-154|https://cwiki.apache.org/confluence/display/KAFKA/KIP-154+Add+Kafka+Connect+configuration+properties+for+creating+internal+topics] / KAFKA-4667 I added a [builder for NewTopic|https://github.com/apache/kafka/pull/2984/files#diff-649c20fa5a9f94e7221bf8ef8211afaeR44] that has methods to set replication factor, num partitions, the cleanup policy (currently compacted only), min ISRs, etc. It's currently an implementation detail of Kafka Connect, but it'd be great if that could move down to the AdminClient. > Review and potentially tweak AdminClient API for the initial release (KIP-117) > -- > > Key: KAFKA-5275 > URL: https://issues.apache.org/jira/browse/KAFKA-5275 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma >Assignee: Ismael Juma > Fix For: 0.11.0.0 > > > Once all the pieces are in, we should take a pass and ensure that the APIs > work well together and that they are consistent. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)
[ https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16015081#comment-16015081 ] Ismael Juma commented on KAFKA-5275: Because the ACLs and Configs PRs were developed in parallel, there's a bit of overlap. We should clean that up too. > Review and potentially tweak AdminClient API for the initial release (KIP-117) > -- > > Key: KAFKA-5275 > URL: https://issues.apache.org/jira/browse/KAFKA-5275 > Project: Kafka > Issue Type: Sub-task >Reporter: Ismael Juma > Fix For: 0.11.0.0 > > > Once all the pieces are in, we should take a pass and ensure that the APIs > work well together and that they are consistent. -- This message was sent by Atlassian JIRA (v6.3.15#6346)