[GitHub] kafka pull request #3639: MINOR: Standardize logging of Worker-level message...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3639 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS] KIP-184 Rename LogCleaner and related classes to LogCompactor

Thanks Ewen for the suggestions. I have updated KIP-184. Updates done are : 1. If deprecated property is encountered currently, then its value will be considered while enabling compactor. 2. log.compactor.min.compaction.lag.ms updated it to be log.compactor.min.lag.ms ( Other naming suggestions

Build failed in Jenkins: kafka-trunk-jdk8 #1893

See Changes: [me] MINOR: Standardize logging of Worker-level messages from Tasks and -- [...truncated 875.49 KB...] kafka.integration.AutoOffsetResetTest >

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

Thanks for the comments, Ewen. Responses inline. > 1. Re: the mention of exactly once, this is within a producer session, > right? And so really only idempotent. Applications still need to take extra > steps for exactly once if they, e.g., are producing data from some other > log like a DB txn

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

>> Another comment about Printed in general is it differs with other options >> that it is a required option than optional one, since it includes toSysOut >> / toFile specs; what are the pros and cons for including these two in the >> option and hence make it a required option than leaving them at

[jira] [Resolved] (KAFKA-5717) [streams] 'null' values in state stores

[ https://issues.apache.org/jira/browse/KAFKA-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy resolved KAFKA-5717. --- Resolution: Fixed Fix Version/s: 0.11.0.1 1.0.0 Issue resolved by pull

[GitHub] kafka pull request #3650: KAFKA-5717: InMemoryKeyValueStore should delete ke...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3650 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS] KIP-184 Rename LogCleaner and related classes to LogCompactor

Yes, as Ismael noted above, I am not fond of this renaming. Keep in mind that the LogCleaner does not only handle compaction. It is possible to configure a cleanup policy of "compact" and "delete," in which case the LogCleaner also handles removal of old segments. Hence the more general LogCleaner

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

Responses inline. > > However, one thing which has not come out of the JIRA discussion is the > > > actual use cases for batch expiry. > > > > There are two usecases I can think of for batch expiry mechanism > > irrespective of how we try to bound the time (batch.expiry.ms or > >

Re: [DISCUSS] KIP-184 Rename LogCleaner and related classes to LogCompactor

Thanks Ismael, Jason for the suggestion. My bad. I should have followed up on mail-list discussion before starting KIP. Apologies. I am relatively new, so I do not know if any confusion was reported in past due to terminology. May be others can chime in. If the old naming is fine with majority

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

>> The key idea is that by using the same function name string for static >> constructor and member functions, users do not need to remember what are >> the differences but can call these functions with any ordering they want, >> and later calls on the same spec will win over early calls. >> >>

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

On Wed, 9 Aug 2017 at 20:00 Guozhang Wang wrote: > >> Another comment about Printed in general is it differs with other > options > >> that it is a required option than optional one, since it includes > toSysOut > >> / toFile specs; what are the pros and cons for including

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

Thanks for the comments Ismael and Jason. Regarding the OutOfOrderSequenceException, it is more likely when you enable idempotence and have acks=1, simply because you have a greater probability of losing acknowledged data with acks=1, and the error code indicates that. The particular scenario is

Re: [DISCUSS] KIP-184 Rename LogCleaner and related classes to LogCompactor

Hey Pranav, Let's see what others think before closing the KIP. If there are strong reasons for the renaming, I would reconsider. As far as deprecating `log.cleaner.enable`, I think it's a good idea and can be done in a separate KIP. Guozhang's suggestion seems reasonable, but I'd just turn it

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

On Wed, Aug 9, 2017 at 1:28 PM Apurva Mehta wrote: > > > There seems to be no relationship with cluster metadata availability or > > > staleness. Expiry is just based on the time since the batch has been > > ready. > > > Please correct me if I am wrong. > > > > > > > I was

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

Thanks for your email Becket. Your observations around using acks=1 and acks=-1 are correct. Do note that getting an OutOfOrderSequence means that acknowledged data has been lost. This could be due to a weaker acks setting like acks=1 or due to a topic which is not configured to handle broker

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jun, I have been thinking about whether it is better to return lag (i.e. HW - LEO) instead of LEO. Note that the lag in the DescribeDirsResponse may be negative if LEO > HW. It will almost always be negative for leader and in-sync replicas. Note that we can not calculate lag as max(0, HW -

Re: [DISCUSS] KIP-184 Rename LogCleaner and related classes to LogCompactor

Hi Pranav, In the JIRA and the previous mailing list thread, some of us wondered if the benefit is worth the transition pain. To quote Jason: "The "log cleaner" naming may not be ideal, but it is not incorrect and some of the terminology used elsewhere makes more sense given this name (e.g.

Jenkins build is back to normal : kafka-trunk-jdk8 #1892

See

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

Apurva, For the benchmarking, I have a couple of questions: 1. Re: the mention of exactly once, this is within a producer session, right? And so really only idempotent. Applications still need to take extra steps for exactly once if they, e.g., are producing data from some other log like a DB

Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days

Thanks for the KIP, +1 from me. Ismael On Wed, Aug 9, 2017 at 1:24 AM, Ewen Cheslack-Postava wrote: > Hi all, > > I posted a simple new KIP for a problem we see with a lot of users: > KIP-186: Increase offsets retention default to 7 days > >

Build failed in Jenkins: kafka-0.11.0-jdk7 #264

See Changes: [me] KAFKA-5704: Corrected Connect distributed startup behavior to allow -- [...truncated 861.89 KB...] kafka.api.SaslScramSslEndToEndAuthorizationTest >

Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days

Yes the current default is too short, +1 On Wed, Aug 9, 2017 at 8:56 AM, Ismael Juma wrote: > Thanks for the KIP, +1 from me. > > Ismael > > On Wed, Aug 9, 2017 at 1:24 AM, Ewen Cheslack-Postava > wrote: > >> Hi all, >> >> I posted a simple new KIP for a

[jira] [Created] (KAFKA-5716) Connect: When SourceTask.commit it is possible not everthing from SourceTask.poll has been sent

Per Steffensen created KAFKA-5716: - Summary: Connect: When SourceTask.commit it is possible not everthing from SourceTask.poll has been sent Key: KAFKA-5716 URL: https://issues.apache.org/jira/browse/KAFKA-5716

[GitHub] kafka pull request #3649: MINOR: Add one more instruction

GitHub user enothereska opened a pull request: https://github.com/apache/kafka/pull/3649 MINOR: Add one more instruction You can merge this pull request into a Git repository by running: $ git pull https://github.com/enothereska/kafka minor-docker-docs Alternatively you can

Re: [DISCUSS] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

Hi Ewen, Thanks for looking at the KIP. I've updated it for some of your comments, but see also a few replies inline. On 9 August 2017 at 06:02, Ewen Cheslack-Postava wrote: > Thanks for the KIP. Generally the move away from ZK and to native Kafka > requests is good, so I'm

[jira] [Created] (KAFKA-5717) [streams] 'null' values in state stores

Bart Vercammen created KAFKA-5717: - Summary: [streams] 'null' values in state stores Key: KAFKA-5717 URL: https://issues.apache.org/jira/browse/KAFKA-5717 Project: Kafka Issue Type: Bug

Re: 答复: [DISCUSS] KIP-178: Size-based log directory selection strategy

Hi Hu, I wonder whether changing, or configuring a size-balancing strategy would be sufficient for all users. I would expect that users might want to take other factors into account. For example, with KIP-113, balancing IO across the disks might also be a factor, in addition to balancing free

Re: [DISCUSS] KIP-174 - Deprecate and remove internal converter configs in WorkerConfig

Thanks Ewen, I just edited the KIP to reflect the changes. Regards, Umesh On Wed, 9 Aug 2017 at 11:00 Ewen Cheslack-Postava wrote: > Great, looking good. I'd probably be a bit more concrete about the > Proposed Changes (e.g., "will log an warning if the config is specified"

Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days

Just had this create issues at a customer as well, +1 On Wed, Aug 9, 2017 at 11:46 AM, Mickael Maison wrote: > Yes the current default is too short, +1 > > On Wed, Aug 9, 2017 at 8:56 AM, Ismael Juma wrote: > > Thanks for the KIP, +1 from me. > > >

Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days

Thanks for the KIP. +1 from me. On Tue, Aug 8, 2017 at 5:24 PM, Ewen Cheslack-Postava wrote: > Hi all, > > I posted a simple new KIP for a problem we see with a lot of users: > KIP-186: Increase offsets retention default to 7 days > >

[GitHub] kafka pull request #3566: KAFKA-5629: Changed behavior of ConsoleConsumer wh...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3566 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [VOTE] KIP-175: Additional '--describe' views for ConsumerGroupCommand

Thanks for the KIP. +1 On Thu, Jul 27, 2017 at 2:04 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi all, > > Thanks to everyone who participated in the discussion on KIP-175, and > provided feedback. > The KIP can be found at >

[GitHub] kafka pull request #3651: MINOR: Add missing deprecations on old request obj...

GitHub user hachikuji opened a pull request: https://github.com/apache/kafka/pull/3651 MINOR: Add missing deprecations on old request objects You can merge this pull request into a Git repository by running: $ git pull https://github.com/hachikuji/kafka

Build failed in Jenkins: kafka-trunk-jdk8 #1894

See Changes: [damian.guy] KAFKA-5717; InMemoryKeyValueStore should delete keys with null values -- [...truncated 3.68 MB...]

Jenkins build is back to normal : kafka-trunk-jdk7 #2622

See

[jira] [Resolved] (KAFKA-5629) Console Consumer overrides auto.offset.reset property when provided on the command line without warning about it.

[ https://issues.apache.org/jira/browse/KAFKA-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-5629. Resolution: Fixed Fix Version/s: 1.0.0 Issue resolved by pull request 3566

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

> > There seems to be no relationship with cluster metadata availability or > > staleness. Expiry is just based on the time since the batch has been > ready. > > Please correct me if I am wrong. > > > > I was not very specific about where we do expiration. I glossed over some > details because

Jenkins build is back to normal : kafka-0.11.0-jdk7 #266

See

Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days

+1 on the bump to 7 days. Wanted to mention one minor point. The OffsetCommit RPC still provides the ability to set the retention time from the client, but we do not use it in the consumer. Should we consider adding a consumer config to set this? Given the problems people had with the old default,

[GitHub] kafka-site pull request #70: Add learn more verbiage on ctas

GitHub user derrickdoo opened a pull request: https://github.com/apache/kafka-site/pull/70 Add learn more verbiage on ctas Added 'learn more' to the 3 CTAs at the very top of the homepage You can merge this pull request into a Git repository by running: $ git pull

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

Minor correction: the OutOfOrderSequenceException is not fatal for the idempotent producer and it is not necessarily tied to the acks setting (though it is more likely to be thrown with acks=1). It is used to signal the user that there was a gap in the delivery of messages. You can hit this if

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

Hi Jason, Thanks for the correction. See inline. On Wed, Aug 9, 2017 at 5:13 PM, Jason Gustafson wrote: > Minor correction: the OutOfOrderSequenceException is not fatal for the > idempotent producer and it is not necessarily tied to the acks setting > (though it is more

Build failed in Jenkins: kafka-trunk-jdk7 #2621

See Changes: [me] MINOR: Standardize logging of Worker-level messages from Tasks and -- [...truncated 1.97 MB...]

Kafka Producer/Consumer APIs vs Apache Camel

Hello All, Are their any advantages of using Kafka Producer/Consumer APIs over Apache Camel? Thanks!! -- NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution

[jira] [Created] (KAFKA-5719) Create a quickstart archetype project for Kafka Streams

Guozhang Wang created KAFKA-5719: Summary: Create a quickstart archetype project for Kafka Streams Key: KAFKA-5719 URL: https://issues.apache.org/jira/browse/KAFKA-5719 Project: Kafka Issue

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

Thanks for the KIP, Apurva. It is a good time to review the configurations to see if we can improve the user experience. We also might need to think from users standpoint about the out of the box experience. 01. Generally speaking, I think it makes sense to make idempotence=true so we can enable

Build failed in Jenkins: kafka-trunk-jdk8 #1895

See -- [...truncated 4.20 MB...] kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow STARTED kafka.integration.AutoOffsetResetTest >

[jira] [Created] (KAFKA-5720) In Jenkins, kafka.api.SaslSslAdminClientIntegrationTest failed with org.apache.kafka.common.errors.TimeoutException

Colin P. McCabe created KAFKA-5720: -- Summary: In Jenkins, kafka.api.SaslSslAdminClientIntegrationTest failed with org.apache.kafka.common.errors.TimeoutException Key: KAFKA-5720 URL:

Usage of SSL_ENDPOINT_IDENTIFICATION_ALGORITHM_CONFIG

Hello, I have been trying to find the usage of this property within Kafka source. All I am trying to understand is how this is looked up for hostname verification. I used notepad++ to search for this and only link was in KafkaConfig.scala file - after that I cannot see this scala property being

[GitHub] kafka pull request #3649: MINOR: Add one more instruction

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3649 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka-site pull request #70: Add learn more verbiage on ctas

Github user asfgit closed the pull request at: https://github.com/apache/kafka-site/pull/70 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka-site issue #70: Add learn more verbiage on ctas

Github user guozhangwang commented on the issue: https://github.com/apache/kafka-site/pull/70 LGTM. Merged to `asf-site`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

Re: [DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

Hi Apurva, Thanks for taking another look. Responses below: On Mon, Aug 7, 2017, at 17:47, Apurva Mehta wrote: > The KIP looks good to me. In your latest proposal, the change of state > would be captured as followed in the metrics for groups using Kafka for > membership management: > >

[GitHub] kafka pull request #3371: KAFKA-5470: Replace -XX:+DisableExplicitGC with -X...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3371 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

Sounds good to me. On Wed, Aug 9, 2017 at 11:32 PM, Colin McCabe wrote: > Hi Apurva, > > Thanks for taking another look. Responses below: > > On Mon, Aug 7, 2017, at 17:47, Apurva Mehta wrote: > > The KIP looks good to me. In your latest proposal, the change of state > >

[GitHub] kafka pull request #2863: KAFKA-5077 fix GC logging arguments for Java 9

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2863 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Resolved] (KAFKA-5077) Make server start script work against Java 9

[ https://issues.apache.org/jira/browse/KAFKA-5077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5077. Resolution: Fixed > Make server start script work against Java 9 >

[GitHub] kafka pull request #3421: KAFKA-5507: Check classpath empty in kafka-run-cla...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3421 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Dong, As for whether to return LEO or lag, my point was the following. What you are concerned about is that an in-sync replica could become out of sync again. However, the more common case is that once a replica is caught up, it will stay in sync afterwards. In that case, once the

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jun, If I understand you right, you are suggesting that, in the case when there is continuous incoming traffic, the approach in the KIP-179 will report lag as 0 whereas the approach using DescribeDirsRequest will report lag as non-zero. But I think the approach in KIP-179 will also report

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Dong, Yes, the lag in a replica is calculated as the difference of LEO of the replica and the HW. So, as long as a replica is in sync, the lag is always 0. So, I was suggesting to return lag instead of LEO in DescribeDirsResponse for each replica. I am not sure if we need to return HW

[GitHub] kafka pull request #3651: MINOR: Add missing deprecations on old request obj...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3651 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jun, It just came to me that you may be assuming that folower_lag = HW - follower_LEO. If that is the case, then we need to have new request/response to retrieve this lag since the DescribeDirsResponse doesn't even include HW. It seems that KIP-179 does not explicitly specify the definition

Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days

Any interest on having offsets.retention.minutes= log.retention.(ms|minutes|hours) as a dynamic setting if not set, and having the option the override to a constant value? That would address different types of deployments as well, who modify the default log retention period On 10/8/17, 5:11

[jira] [Created] (KAFKA-5721) Kafka broker stops after network failure

Val Feron created KAFKA-5721: Summary: Kafka broker stops after network failure Key: KAFKA-5721 URL: https://issues.apache.org/jira/browse/KAFKA-5721 Project: Kafka Issue Type: Bug

Jenkins build is back to normal : kafka-trunk-jdk8 #1896

See

Re: [DISCUSS] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

On Wed, Aug 9, 2017 at 11:40 AM, Tom Bentley wrote: > > There are responses with detailed error messages as well as the codes, > CreateTopicsResponse, {Describe|Alter}ConfigsResponse, and the responses > for managing ACLs for instance. To be honest, I assumed including a

[GitHub] kafka pull request #3633: MINOR: streams memory management docs

Github user dguy closed the pull request at: https://github.com/apache/kafka/pull/3633 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

Thanks for the KIP, Apurva. In general, I think it's a good idea to strengthen the guarantees we provide by default. And people who are willing to trade correctness for performance can then change the configs to suit them. I will comment on the KIP specifics in more detail later, but one

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi Dong and Jun, Thanks for your responses! Jun's interpretation of how AlterTopicsRequest could be sent to any broker is indeed what I meant. Since the data has to get persisted in ZK anyway, it doesn't really matter whether we send it to the controller (it will will have to write it to the

[jira] [Created] (KAFKA-5718) Better document what LogAppendTime means

Dustin Cote created KAFKA-5718: -- Summary: Better document what LogAppendTime means Key: KAFKA-5718 URL: https://issues.apache.org/jira/browse/KAFKA-5718 Project: Kafka Issue Type: Improvement

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

On Tue, 8 Aug 2017 at 20:11 Guozhang Wang wrote: > Damian, > > Thanks for the proposal, I had a few comments on the APIs: > > 1. Printed#withFile seems not needed, as users should always spec if it is > to sysOut or to File at the beginning. In addition as a second thought, I

[GitHub] kafka pull request #3650: KAFKA-5717: InMemoryKeyValueStore should delete ke...

GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/3650 KAFKA-5717: InMemoryKeyValueStore should delete keys with null values during restore Fixed a bug in the InMemoryKeyValueStore restoration where a key with a `null` value is written in to the map

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

Hi Dong and Jun, Thanks again for your input in this discussion and on KIP-113. It's difficult that discussion is split between this thread and the one for KIP-113, but I'll try to respond on this thread to questions asked on this thread. It seems there is some consensus that the alterTopic()

[jira] [Resolved] (KAFKA-2360) The kafka-consumer-perf-test.sh script help information print useless parameters.

[ https://issues.apache.org/jira/browse/KAFKA-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-2360. Resolution: Fixed Fix Version/s: 1.0.0 > The kafka-consumer-perf-test.sh script help

Re: [DISCUSS] KAFKA-4930 & KAFKA 4938 - Treatment of name parameter in create connector requests

Could someone have a look at the PR for KAFKA-4930 if they get the chance (not necessarily you Gwen, just bumping in general)? I've updated it according to the latest comments a little while ago and would like to get this done, before I forget what I did in case more changes are necessary :)

Build failed in Jenkins: kafka-0.11.0-jdk7 #265

See Changes: [damian.guy] MINOR: streams memory management docs -- [...truncated 2.42 MB...] org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStoreTest