date:20171221

Re: [VOTE] Allowing write access to GitHub repositories (aka GitBox)

2017-12-21 Thread Guozhang Wang

Thanks Ismael,

I have setup the account and can verify that it works on kafka / kafka-site
PRs now!


Guozhang

On Thu, Dec 21, 2017 at 5:12 PM, Ismael Juma  wrote:

> The previous Apache git repos don't exist anymore. The new ones are at:
>
> https://gitbox.apache.org/repos/asf/?p=kafka.git
> https://gitbox.apache.org/repos/asf/?p=kafka-site.git
>
> In any case, we should merge to the GitHub repo now and the following PR
> updates the merge script to default to that:
>
> https://github.com/apache/kafka/pull/4352
>
> Ismael
>
> On Fri, Dec 22, 2017 at 1:02 AM, Ismael Juma  wrote:
>
> > It's done! Committers, please set up your account:
> >
> > https://gitbox.apache.org/setup/
> >
> > On Wed, Dec 20, 2017 at 12:30 AM, Ismael Juma  wrote:
> >
> >> Forgot the link to the relevant Infra JIRA: https://issues.apache.or
> >> g/jira/browse/INFRA-15676
> >>
> >> On Tue, Dec 19, 2017 at 11:59 PM, Ismael Juma 
> wrote:
> >>
> >>> GitBox migration will happen today. Committers, please make sure to
> >>> associate your github ID with your apache.org account via
> id.apache.org,
> >>> and make sure to enable 2 factor authentication in GitHub.
> >>>
> >>> Ismael
> >>>
> >>> On Fri, Dec 15, 2017 at 3:40 PM, Ismael Juma 
> wrote:
> >>>
>  Thanks to everyone who voted and contributed to the discussion.
> 
>  The vote passes with 7 binding votes (Damian, Rajini, Jason, Gwen,
>  Guozhang, Sriram, Ismael) and 2 non-binding votes (Manikumar and Tom).
> 
>  I will file a JIRA ticket in the Apache Infra project requesting the
>  migration to GitBox.
> 
>  Ismael
> 
>  On Thu, Dec 14, 2017 at 11:48 AM, Tom Bentley 
>  wrote:
> 
> > +1
> >
> > On 12 December 2017 at 20:38, Sriram Subramanian 
> > wrote:
> >
> > > +1
> > >
> > > On Tue, Dec 12, 2017 at 8:22 AM, Manikumar <
> > manikumar.re...@gmail.com>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > On Tue, Dec 12, 2017 at 9:49 PM, Rajini Sivaram <
> > rajinisiva...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Thanks, Ismael!
> > > > >
> > > > > On Tue, Dec 12, 2017 at 4:18 PM, Damian Guy <
> > damian@gmail.com>
> > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > On Tue, 12 Dec 2017 at 15:47 Ismael Juma 
> > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > The Apache Infra team has started a new project earlier
> this
> > year
> > > > > called
> > > > > > > GitBox that supports two-way synchronization between GitHub
> > and
> > > > > > > git-wip-us.apache.org and, most importantly, provides
> > GitHub write
> > > > > > access
> > > > > > > to committers. GitBox is not generally available yet, but
> > > individual
> > > > > > > projects can ask to be migrated.
> > > > > > >
> > > > > > > I would like to start a vote on migrating kafka and
> > kafka-site to
> > > > > GitBox
> > > > > > > and:
> > > > > > >
> > > > > > > 1. Providing GitHub write access to committers (this
> > requires dual
> > > > > factor
> > > > > > > authentication)
> > > > > > > 2. Allowing merges via the GitHub UI as well as the
> existing
> > merge
> > > > > script
> > > > > > > 3. Enabling protected branches for trunk and release
> > branches so
> > > that
> > > > > > > merges via the GitHub UI can only be done if the tests pass
> > and the
> > > > PR
> > > > > > has
> > > > > > > been approved by a committer
> > > > > > > 4. Only allowing the "squash and merge" strategy for GitHub
> > UI
> > > merges
> > > > > > > 5. Updating the merge script so that the GitHub git repo is
> > the
> > > > target
> > > > > of
> > > > > > > the merge
> > > > > > > 6. Disallowing force pushes to trunk and release branches
> > > > > > >
> > > > > > > The discussion thread talks about some of the pros and cons
> > (mostly
> > > > > pros)
> > > > > > > of this change:
> > > > > > >
> > > > > > >
> > > > > > > https://lists.apache.org/thread.html/
> > > 7031168e7026222169c66fed29f520
> > > > > > 0fc4b561df28c242ccf706f326@%3Cdev.kafka.apache.org%3E
> > > > > > >
> > > > > > > The vote will run for 72 hours.
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 
> 
> >>>
> >>
> >
>



-- 
-- Guozhang

[jira] [Resolved] (KAFKA-3496) Add reconnect attemps policies for client

2017-12-21 Thread Ismael Juma (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-3496.

Resolution: Won't Fix

We decided to add configs for enabling exponential backoff instead.

> Add reconnect attemps policies for client
> -
>
> Key: KAFKA-3496
> URL: https://issues.apache.org/jira/browse/KAFKA-3496
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.10.0.0
>Reporter: Florian Hussonnois
>
> Currently the client reconnection attempts is only controlled by the property 
> : reconnect.backoff.ms
> It would be nice to introduce a reconnect attempt policy. At first, two 
> policies may be defined : 
> - ConstantReconnectAttemptPolicy
> - ExponentialReconnectAttemptPolicy
> The policy could be then configure as follows : 
> Properties config = new Properties(); 
> config.put(ConsumerConfig.RECONNECT_ATTEMPTS_POLICY_CLASS_CONFIG, 
> "org.apache.kafka.clients.ExponentialReconnectAttemptPolicy");
> config.put(ConsumerConfig.RECONNECT_EXPONENTIAL_MAX_DELAY_MS_CONFIG, 5000);



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-5895) Gradle 3.0+ is needed on the build

2017-12-21 Thread Ismael Juma (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-5895.

   Resolution: Fixed
Fix Version/s: 1.1.0

> Gradle 3.0+ is needed on the build
> --
>
> Key: KAFKA-5895
> URL: https://issues.apache.org/jira/browse/KAFKA-5895
> Project: Kafka
>  Issue Type: Improvement
>  Components: documentation
>Affects Versions: 0.11.0.2
>Reporter: Matthias Weßendorf
>Priority: Trivial
> Fix For: 1.1.0
>
>
> The README says:
> Kafka requires Gradle 2.0 or higher.
> but running with "2.13", I am getting an ERROR message, saying that 3.0+ is 
> needed:
> {code}
> > Failed to apply plugin [class 
> > 'com.github.jengelman.gradle.plugins.shadow.ShadowBasePlugin']
>> This version of Shadow supports Gradle 3.0+ only. Please upgrade.
> {code}
> Full log here:
> {code}
> ➜  kafka git:(utils_improvment) ✗ gradle 
> To honour the JVM settings for this build a new JVM will be forked. Please 
> consider using the daemon: 
> https://docs.gradle.org/2.13/userguide/gradle_daemon.html.
> Download 
> https://jcenter.bintray.com/org/ajoberstar/grgit/1.9.3/grgit-1.9.3.pom
> Download 
> https://jcenter.bintray.com/com/github/ben-manes/gradle-versions-plugin/0.15.0/gradle-versions-plugin-0.15.0.pom
> Download 
> https://repo1.maven.org/maven2/org/scoverage/gradle-scoverage/2.1.0/gradle-scoverage-2.1.0.pom
> Download 
> https://jcenter.bintray.com/com/github/jengelman/gradle/plugins/shadow/2.0.1/shadow-2.0.1.pom
> Download 
> https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit/4.5.2.201704071617-r/org.eclipse.jgit-4.5.2.201704071617-r.pom
> Download 
> https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit-parent/4.5.2.201704071617-r/org.eclipse.jgit-parent-4.5.2.201704071617-r.pom
> Download 
> https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit.ui/4.5.2.201704071617-r/org.eclipse.jgit.ui-4.5.2.201704071617-r.pom
> Download 
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.jsch/0.0.9/jsch.agentproxy.jsch-0.0.9.pom
> Download 
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy/0.0.9/jsch.agentproxy-0.0.9.pom
> Download 
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.pageant/0.0.9/jsch.agentproxy.pageant-0.0.9.pom
> Download 
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.sshagent/0.0.9/jsch.agentproxy.sshagent-0.0.9.pom
> Download 
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.usocket-jna/0.0.9/jsch.agentproxy.usocket-jna-0.0.9.pom
> Download 
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.usocket-nc/0.0.9/jsch.agentproxy.usocket-nc-0.0.9.pom
> Download https://repo1.maven.org/maven2/com/jcraft/jsch/0.1.54/jsch-0.1.54.pom
> Download 
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.core/0.0.9/jsch.agentproxy.core-0.0.9.pom
> Download 
> https://jcenter.bintray.com/org/ajoberstar/grgit/1.9.3/grgit-1.9.3.jar
> Download 
> https://jcenter.bintray.com/com/github/ben-manes/gradle-versions-plugin/0.15.0/gradle-versions-plugin-0.15.0.jar
> Download 
> https://repo1.maven.org/maven2/org/scoverage/gradle-scoverage/2.1.0/gradle-scoverage-2.1.0.jar
> Download 
> https://jcenter.bintray.com/com/github/jengelman/gradle/plugins/shadow/2.0.1/shadow-2.0.1.jar
> Download 
> https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit/4.5.2.201704071617-r/org.eclipse.jgit-4.5.2.201704071617-r.jar
> Download 
> https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit.ui/4.5.2.201704071617-r/org.eclipse.jgit.ui-4.5.2.201704071617-r.jar
> Download 
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.jsch/0.0.9/jsch.agentproxy.jsch-0.0.9.jar
> Download 
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.pageant/0.0.9/jsch.agentproxy.pageant-0.0.9.jar
> Download 
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.sshagent/0.0.9/jsch.agentproxy.sshagent-0.0.9.jar
> Download 
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.usocket-jna/0.0.9/jsch.agentproxy.usocket-jna-0.0.9.jar
> Download 
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.usocket-nc/0.0.9/jsch.agentproxy.usocket-nc-0.0.9.jar
> Download 
> https://repo1.maven.org/maven2/com/jcraft/jsch.agentproxy.core/0.0.9/jsch.agentproxy.core-0.0.9.jar
> Building project 'core' with Scala version 2.11.11
> FAILURE: Build failed with an exception.
> * Where:
> Build file '/home/Matthias/Work/Apache/kafka/build.gradle' line: 978
> * What went wrong:
> A problem occurred evaluating root project 'kafka'.
> > Failed to apply plugin [class 
> > 'com.github.jengelman.gradle.plugins.shadow.ShadowBasePlugin']
>> This version of Shadow supports Gradle 3.0+ only. Please upgrade.
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or 
> --debug

Re: [VOTE] Allowing write access to GitHub repositories (aka GitBox)

2017-12-21 Thread Ismael Juma

The previous Apache git repos don't exist anymore. The new ones are at:

https://gitbox.apache.org/repos/asf/?p=kafka.git
https://gitbox.apache.org/repos/asf/?p=kafka-site.git

In any case, we should merge to the GitHub repo now and the following PR
updates the merge script to default to that:

https://github.com/apache/kafka/pull/4352

Ismael

On Fri, Dec 22, 2017 at 1:02 AM, Ismael Juma  wrote:

> It's done! Committers, please set up your account:
>
> https://gitbox.apache.org/setup/
>
> On Wed, Dec 20, 2017 at 12:30 AM, Ismael Juma  wrote:
>
>> Forgot the link to the relevant Infra JIRA: https://issues.apache.or
>> g/jira/browse/INFRA-15676
>>
>> On Tue, Dec 19, 2017 at 11:59 PM, Ismael Juma  wrote:
>>
>>> GitBox migration will happen today. Committers, please make sure to
>>> associate your github ID with your apache.org account via id.apache.org,
>>> and make sure to enable 2 factor authentication in GitHub.
>>>
>>> Ismael
>>>
>>> On Fri, Dec 15, 2017 at 3:40 PM, Ismael Juma  wrote:
>>>
 Thanks to everyone who voted and contributed to the discussion.

 The vote passes with 7 binding votes (Damian, Rajini, Jason, Gwen,
 Guozhang, Sriram, Ismael) and 2 non-binding votes (Manikumar and Tom).

 I will file a JIRA ticket in the Apache Infra project requesting the
 migration to GitBox.

 Ismael

 On Thu, Dec 14, 2017 at 11:48 AM, Tom Bentley 
 wrote:

> +1
>
> On 12 December 2017 at 20:38, Sriram Subramanian 
> wrote:
>
> > +1
> >
> > On Tue, Dec 12, 2017 at 8:22 AM, Manikumar <
> manikumar.re...@gmail.com>
> > wrote:
> >
> > > +1
> > >
> > > On Tue, Dec 12, 2017 at 9:49 PM, Rajini Sivaram <
> rajinisiva...@gmail.com
> > >
> > > wrote:
> > >
> > > > +1
> > > >
> > > > Thanks, Ismael!
> > > >
> > > > On Tue, Dec 12, 2017 at 4:18 PM, Damian Guy <
> damian@gmail.com>
> > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > On Tue, 12 Dec 2017 at 15:47 Ismael Juma 
> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > The Apache Infra team has started a new project earlier this
> year
> > > > called
> > > > > > GitBox that supports two-way synchronization between GitHub
> and
> > > > > > git-wip-us.apache.org and, most importantly, provides
> GitHub write
> > > > > access
> > > > > > to committers. GitBox is not generally available yet, but
> > individual
> > > > > > projects can ask to be migrated.
> > > > > >
> > > > > > I would like to start a vote on migrating kafka and
> kafka-site to
> > > > GitBox
> > > > > > and:
> > > > > >
> > > > > > 1. Providing GitHub write access to committers (this
> requires dual
> > > > factor
> > > > > > authentication)
> > > > > > 2. Allowing merges via the GitHub UI as well as the existing
> merge
> > > > script
> > > > > > 3. Enabling protected branches for trunk and release
> branches so
> > that
> > > > > > merges via the GitHub UI can only be done if the tests pass
> and the
> > > PR
> > > > > has
> > > > > > been approved by a committer
> > > > > > 4. Only allowing the "squash and merge" strategy for GitHub
> UI
> > merges
> > > > > > 5. Updating the merge script so that the GitHub git repo is
> the
> > > target
> > > > of
> > > > > > the merge
> > > > > > 6. Disallowing force pushes to trunk and release branches
> > > > > >
> > > > > > The discussion thread talks about some of the pros and cons
> (mostly
> > > > pros)
> > > > > > of this change:
> > > > > >
> > > > > >
> > > > > > https://lists.apache.org/thread.html/
> > 7031168e7026222169c66fed29f520
> > > > > 0fc4b561df28c242ccf706f326@%3Cdev.kafka.apache.org%3E
> > > > > >
> > > > > > The vote will run for 72 hours.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > >
> > > >
> > >
> >
>


>>>
>>
>

Re: [VOTE] Allowing write access to GitHub repositories (aka GitBox)

2017-12-21 Thread Ismael Juma

It's done! Committers, please set up your account:

https://gitbox.apache.org/setup/

On Wed, Dec 20, 2017 at 12:30 AM, Ismael Juma  wrote:

> Forgot the link to the relevant Infra JIRA: https://issues.apache.
> org/jira/browse/INFRA-15676
>
> On Tue, Dec 19, 2017 at 11:59 PM, Ismael Juma  wrote:
>
>> GitBox migration will happen today. Committers, please make sure to
>> associate your github ID with your apache.org account via id.apache.org,
>> and make sure to enable 2 factor authentication in GitHub.
>>
>> Ismael
>>
>> On Fri, Dec 15, 2017 at 3:40 PM, Ismael Juma  wrote:
>>
>>> Thanks to everyone who voted and contributed to the discussion.
>>>
>>> The vote passes with 7 binding votes (Damian, Rajini, Jason, Gwen,
>>> Guozhang, Sriram, Ismael) and 2 non-binding votes (Manikumar and Tom).
>>>
>>> I will file a JIRA ticket in the Apache Infra project requesting the
>>> migration to GitBox.
>>>
>>> Ismael
>>>
>>> On Thu, Dec 14, 2017 at 11:48 AM, Tom Bentley 
>>> wrote:
>>>
 +1

 On 12 December 2017 at 20:38, Sriram Subramanian 
 wrote:

 > +1
 >
 > On Tue, Dec 12, 2017 at 8:22 AM, Manikumar 
 > wrote:
 >
 > > +1
 > >
 > > On Tue, Dec 12, 2017 at 9:49 PM, Rajini Sivaram <
 rajinisiva...@gmail.com
 > >
 > > wrote:
 > >
 > > > +1
 > > >
 > > > Thanks, Ismael!
 > > >
 > > > On Tue, Dec 12, 2017 at 4:18 PM, Damian Guy 
 > > wrote:
 > > >
 > > > > +1
 > > > >
 > > > > On Tue, 12 Dec 2017 at 15:47 Ismael Juma 
 wrote:
 > > > >
 > > > > > Hi all,
 > > > > >
 > > > > > The Apache Infra team has started a new project earlier this
 year
 > > > called
 > > > > > GitBox that supports two-way synchronization between GitHub
 and
 > > > > > git-wip-us.apache.org and, most importantly, provides GitHub
 write
 > > > > access
 > > > > > to committers. GitBox is not generally available yet, but
 > individual
 > > > > > projects can ask to be migrated.
 > > > > >
 > > > > > I would like to start a vote on migrating kafka and
 kafka-site to
 > > > GitBox
 > > > > > and:
 > > > > >
 > > > > > 1. Providing GitHub write access to committers (this requires
 dual
 > > > factor
 > > > > > authentication)
 > > > > > 2. Allowing merges via the GitHub UI as well as the existing
 merge
 > > > script
 > > > > > 3. Enabling protected branches for trunk and release branches
 so
 > that
 > > > > > merges via the GitHub UI can only be done if the tests pass
 and the
 > > PR
 > > > > has
 > > > > > been approved by a committer
 > > > > > 4. Only allowing the "squash and merge" strategy for GitHub UI
 > merges
 > > > > > 5. Updating the merge script so that the GitHub git repo is
 the
 > > target
 > > > of
 > > > > > the merge
 > > > > > 6. Disallowing force pushes to trunk and release branches
 > > > > >
 > > > > > The discussion thread talks about some of the pros and cons
 (mostly
 > > > pros)
 > > > > > of this change:
 > > > > >
 > > > > >
 > > > > > https://lists.apache.org/thread.html/
 > 7031168e7026222169c66fed29f520
 > > > > 0fc4b561df28c242ccf706f326@%3Cdev.kafka.apache.org%3E
 > > > > >
 > > > > > The vote will run for 72 hours.
 > > > > >
 > > > > > Ismael
 > > > > >
 > > > >
 > > >
 > >
 >

>>>
>>>
>>
>

Re: [VOTE] KIP-237: More Controller Health Metrics

2017-12-21 Thread Dong Lin

Bump up the thread so that we can have these sensors to monitor our Kafka
service sooner.

On Mon, Dec 18, 2017 at 2:03 PM, Dong Lin  wrote:

> Hi all,
>
> Since there are no more outstanding comments, I would like to start voting
> thread for KIP-237: https://cwiki.apache.org/confluence/display/KAFKA/
> KIP-237%3A+More+Controller+Health+Metrics
>
> The KIP proposes to add a few more metrics to help monitor Kafka
> Controller health.
>
> Thanks,
> Dong
>

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

2017-12-21 Thread Dong Lin

Hey Jun,

Thanks much for your comments. Yeah I have not considered the case where
the offset is stored externally.

Based Jason's question, I think we probably have to use a global
metadata_epoch. And since we have a global metadata_epoch, this KIP
probably no longer needs the per-partition leader_epoch. Then we can use
two newly-added API in consumer that allows user to get the metadata_epoch
from consumer and wait for consumer to receive MetadataResponse whose
metadata_epoch >= the given metadata_epoch. These two APIs should address
the case where user stored offset externally. I have updated the KIP
accordingly. Could you take another look?

Thanks for all the comments.

Dong


On Tue, Dec 19, 2017 at 3:09 PM, Jun Rao  wrote:

> Hi, Dong,
>
> Thanks for the reply.
>
> 10. I was actually just thinking the case when the consumer consumes old
> data. If the current leader epoch is 3 and the consumer is consuming
> records generated in leader epoch 1, the epoch associated with the offset
> should be 1. However, as you pointed out, the fetch response currently
> includes the leader epoch for fetched data. So, this is already covered.
>
> 11. That's an interesting thought. What about the case when the offsets are
> stored externally? When we restart a consumer and seek to an externally
> stored offset, we won't know the leader epoch in the consumer. Do we need
> another request to retrieve the leader epoch based on an offset and make
> sure the info is up to date? Another related thing is that the leader epoch
> that we want to associate the offset with ideally should be the epoch when
> the data is fetched. For example, when all replicas lost data due to a
> power failure or when there is an unclean leader election, the leader epoch
> for a given offset may change over time on the broker. In those cases, a
> consumer's offset may be in range, but is not in the same leader epoch for
> the time when the data is fetched. We can potentially do a smarter offset
> reset in those cases if we remember the epoch when the data is fetched.
>
> Jun
>
>
>
> On Mon, Dec 18, 2017 at 1:58 PM, Dong Lin  wrote:
>
> > Hey Jun,
> >
> > Thanks much for your comments. These are very thoughtful ideas. Please
> see
> > my comments below.
> >
> > On Thu, Dec 14, 2017 at 6:38 PM, Jun Rao  wrote:
> >
> > > Hi, Dong,
> > >
> > > Thanks for the update. A few more comments below.
> > >
> > > 10. It seems that we need to return the leader epoch in the fetch
> > response
> > > as well When fetching data, we could be fetching data from a leader
> epoch
> > > older than what's returned in the metadata response. So, we want to use
> > the
> > > leader epoch associated with the offset being fetched for committing
> > > offsets.
> > >
> >
> > It seems that we may have two separate issues here. The first issue is
> that
> > consumer uses metadata that is older than the one it uses before. The
> > second issue is that consumer uses metadata which is newer than the
> > corresponding leader epoch in the leader broker. We know that the
> > OffsetOutOfRangeException described in this KIP can be prevented by
> > avoiding the first issue. On the other hand, it seems that the
> > OffsetOffsetOutOfRangeException can still happen even if we avoid the
> > second issue -- if consumer uses an older version of metadata, the leader
> > epoch in its metadata may equal the leader epoch in the broker even if
> the
> > leader epoch in the broker is oudated.
> >
> > Given this understanding, I am not sure why we need to return the leader
> > epoch in the fetch response. As long as consumer's metadata is not going
> > back in version, I think we are good. Did I miss something here?
> >
> >
> > >
> > > 11. Should we now extend OffsetAndMetadata used in the offset commit
> api
> > in
> > > KafkaConsumer to include leader epoch? Similarly, should we return
> leader
> > > epoch in endOffsets(), beginningOffsets() and position()? We probably
> > need
> > > to think about how to make the api backward compatible.
> > >
> >
> > After thinking through this carefully, I think we probably don't want to
> > extend OffsetAndMetadata to include leader epoch because leader epoch is
> > kind of implementation detail which ideally should be hidden from user.
> The
> > consumer can include leader epoch in the OffsetCommitRequest after taking
> > offset from commitSync(final Map
> > offsets). Similarly consumer can store leader epoch from
> > OffsetFetchResponse and only provide offset to user via
> > consumer.committed(topicPartition). This solution seems to work well and
> > we
> > don't have to make changes to consumer's public API. Does this sound OK?
> >
> >
> > >
> > > 12. It seems that we now need to store leader epoch in the offset
> topic.
> > > Could you include the new schema for the value of the offset topic and
> > add
> > > upgrade notes?
> >
> >
> > You are right. I have

[GitHub] kafka pull request #4351: kafka-6320: move ZK metrics in KafkaHealthCheck to...

2017-12-21 Thread junrao

GitHub user junrao opened a pull request:

https://github.com/apache/kafka/pull/4351

kafka-6320: move ZK metrics in KafkaHealthCheck to ZookeeperClient

* Moved metrics in KafkaHealthCheck to ZookeeperClient.
* Converted remaining ZkUtils usage in KafkaServer to ZookeeperClient and 
removed ZkUtils from KafkaServer.
* Made the re-creation of ZooKeeper during ZK session expiration with 
infinite retries.
* Added unit tests for all new methods in KafkaZkClient.

### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation 
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/junrao/kafka kafka-6320

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4351.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4351


commit 68937522ada25ed1efdda0f51919375d997e596e
Author: Jun Rao 
Date:   2017-12-21T22:09:17Z

kafka-6320: move ZK metrics in KafkaHealthCheck to ZookeeperClient




---

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

2017-12-21 Thread Dong Lin

Hey Jason,

Thanks much. Great question. I have considered topic deletion but I have
not considered the scenario that user creates topic very soon after topic
deletion.

After thinking through this scenario, I think the only option is to have a
global metadata_epoch that keeps increasing every time controller sends
UpdateMetadataRequest. Any other per-topic or per-partition epoch field
will not survive topic deletion followed by topic creation. I have updated
the KIP to use a new design to address all previous questions from you and
Jun. Can you take another look?

Thanks,
Dong

On Tue, Dec 19, 2017 at 2:15 PM, Jason Gustafson  wrote:

> Hey Dong,
>
> One more thought came to mind. Have you considered edge cases around topic
> deletion? I think currently if a topic is deleted and then re-created, the
> leader epoch will start back at the beginning. It seems like that could
> cause trouble for this solution. One thing that helps is that we have logic
> to remove committed offsets for deleted topics, but there may not be any
> guarantees on when that happens relative to when the metadata is updated on
> all brokers. It seems like it could even happen that the topic is deleted
> and recreated quickly enough that the consumer doesn't even "witness" the
> deletion.
>
> Thanks,
> Jason
>
> On Tue, Dec 19, 2017 at 11:40 AM, Jason Gustafson 
> wrote:
>
> > I think you're saying that depending on the bug, in the worst case, you
> > may have to downgrade the client. I think that's fair. Note that one
> > advantage of making this a fatal error is that we'll be more likely to
> hit
> > unexpected edge cases in system tests.
> >
> > -Jason
> >
> > On Tue, Dec 19, 2017 at 11:26 AM, Dong Lin  wrote:
> >
> >> Hey Jason,
> >>
> >> Yeah this may sound a bit confusing. Let me explain my thoughts.
> >>
> >> If there is no bug in the client library, after consumer rebalance or
> >> consumer restart, consume will fetch the previously committed offset and
> >> fetch the committed metadata until the leader epoch in the metadata >=
> the
> >> leader epoch in the OffsetFetchResponse. Therefore, when consumer
> commits
> >> offset later, the leader epoch in the OffsetCommitRequest should be
> larger
> >> than the leader epoch from the previously committed offset. Does this
> >> sound
> >> correct?
> >>
> >> Given the above understanding, it seems to suggest that the only
> >> explanation for this exception is that there is bug in the client
> library.
> >> And due to this specific bug, I am not sure we can avoid this error by
> >> simply restarting consumer. And because this error is non-retriable,
> user
> >> may be forced to downgrade client library. Did I miss something here?
> >>
> >> Thanks,
> >> Dong
> >>
> >>
> >> On Tue, Dec 19, 2017 at 11:19 AM, Jason Gustafson 
> >> wrote:
> >>
> >> > Hey Dong,
> >> >
> >> > Thanks for the updates. Just one question:
> >> >
> >> > When application receives
> >> > > this exception, the only choice will be to revert Kafka client
> >> library to
> >> > > an earlier version.
> >> >
> >> >
> >> > Not sure I follow this. Wouldn't we just restart the consumer? That
> >> would
> >> > cause it to fetch the previous committed offset and then fetch the
> >> correct
> >> > metadata.
> >> >
> >> > Thanks,
> >> > Jason
> >> >
> >> > On Tue, Dec 19, 2017 at 10:36 AM, Dong Lin 
> wrote:
> >> >
> >> > > Hey Jason,
> >> > >
> >> > > Thanks for the comments. These make sense. I have updated the KIP to
> >> > > include a new error INVALID_LEADER_EPOCH. This will be a
> non-retriable
> >> > > error which may be thrown from consumer's API. When application
> >> receives
> >> > > this exception, the only choice will be to revert Kafka client
> >> library to
> >> > > an earlier version.
> >> > >
> >> > > Previously I think it may be better to simply log an error because I
> >> am
> >> > not
> >> > > sure it is a good idea to force user to downgrade Kafka client
> library
> >> > when
> >> > > the error itself, e.g. smaller leader epoch, may not be that fatal.
> >> One
> >> > the
> >> > > other hand it could be argued that we don't know what else can go
> >> wrong
> >> > in
> >> > > the buggy client library and it may be a good reason to force user
> to
> >> > > downgrade library.
> >> > >
> >> > > Thanks,
> >> > > Dong
> >> > >
> >> > >
> >> > > On Tue, Dec 19, 2017 at 9:06 AM, Jason Gustafson <
> ja...@confluent.io>
> >> > > wrote:
> >> > >
> >> > > > Hey Dong,
> >> > > >
> >> > > >
> >> > > > > I think it is a good idea to let coordinator do the additional
> >> sanity
> >> > > > check
> >> > > > > to ensure the leader epoch from OffsetCommitRequest never
> >> decreases.
> >> > > This
> >> > > > > can help us detect bug. The next question will be what should we
> >> do
> >> > if
> >> > > > > OffsetCommitRequest provides a smaller leader epoch. One
> possible
> >> > > > solution
> >> > > > > is to return a

Re: [VOTE] KIP-243: Make ProducerConfig and ConsumerConfig constructors public

2017-12-21 Thread Jason Gustafson

I didn't sense much resistance in that thread, just an effort to keep the
streams and core client config APIs consistent ;).

I'd prefer seeing a KIP for a more general improvement, but this change
seems harmless and improves consistency between the clients, so +1 from me.

-Jason

On Thu, Dec 21, 2017 at 11:19 AM, Matthias J. Sax 
wrote:

> I personally love the builder pattern idea. There was some push back in
> the past though from some people.
>
> cf https://issues.apache.org/jira/browse/KAFKA-4436
>
> Happy to propose the builder pattern but than we should have a proper
> DISCUSS thread. Maybe we do this as a follow up and just do this KIP as-is?
>
>
> -Matthias
>
> On 12/21/17 10:28 AM, Jason Gustafson wrote:
> > Hey Matthias,
> >
> > Let me suggest an alternative. As you have mentioned, these config
> classes
> > do not give users much benefit currently. Maybe we change that? I think
> > many users would appreciate having a builder for configuration since it
> > provides type safety and is generally a much friendlier pattern to work
> > with programmatically. Users could then do something like this:
> >
> > ConsumerConfig config = ConsumerConfig.newBuilder()
> > .setBootstrapServers("localhost:9092")
> > .setGroupId("group")
> > .setRequestTimeout(15, TimeUnit.SECONDS)
> > .build();
> >
> > Consumer consumer = new KafkaConsumer(config);
> >
> > An additional benefit of this is that it gives us a better way to expose
> > config deprecations. In any case, it would make it less odd to expose the
> > public constructor without giving users anything useful to do with the
> > class.
> >
> > What do you think?
> >
> > -Jason
> >
> > On Wed, Dec 20, 2017 at 5:59 PM, Matthias J. Sax 
> > wrote:
> >
> >> It's tailored for internal usage. I think client constructors don't
> >> benefit from accepting those config objects. We just want to be able to
> >> access the default values for certain parameters.
> >>
> >> From a user point of view, it's actually boiler plate code if you pass
> >> in a config object instead of a plain Properties object because the
> >> config object itself is immutable.
> >>
> >> I actually create a JIRA to remove the constructors from KafkaStreams
> >> that do accept StreamsConfig for exact this reason:
> >> https://issues.apache.org/jira/browse/KAFKA-6386
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 12/20/17 3:33 PM, Jason Gustafson wrote:
> >>> Hi Matthias,
> >>>
> >>> Isn't it a little weird to make these constructors public but not also
> >>> expose the corresponding client constructors that use them?
> >>>
> >>> -Jason
> >>>
> >>> On Tue, Dec 19, 2017 at 9:30 AM, Bill Bejeck 
> wrote:
> >>>
>  +1
> 
>  On Tue, Dec 19, 2017 at 12:09 PM, Guozhang Wang 
>  wrote:
> 
> > +1
> >
> > On Tue, Dec 19, 2017 at 1:49 AM, Tom Bentley 
> > wrote:
> >
> >> +1
> >>
> >> On 18 December 2017 at 23:28, Vahid S Hashemian <
> > vahidhashem...@us.ibm.com
> >>>
> >> wrote:
> >>
> >>> +1
> >>>
> >>> Thanks for the KIP.
> >>>
> >>> --Vahid
> >>>
> >>>
> >>>
> >>> From:   Ted Yu 
> >>> To: dev@kafka.apache.org
> >>> Date:   12/18/2017 02:45 PM
> >>> Subject:Re: [VOTE] KIP-243: Make ProducerConfig and
> >> ConsumerConfig
> >>> constructors public
> >>>
> >>>
> >>>
> >>> +1
> >>>
> >>> nit: via "copy and past" an 'e' is missing at the end.
> >>>
> >>> On Mon, Dec 18, 2017 at 2:38 PM, Matthias J. Sax <
> > matth...@confluent.io>
> >>> wrote:
> >>>
>  Hi,
> 
>  I want to propose the following KIP:
> 
> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
> >>> apache.org_confluence_display_KAFKA_KIP-2D=DwIBaQ=jf_
> >>> iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
> >>> kjJc7uSVcviKUc=JToRX4-HeVsRoOekIz18ht-YLMe-T21MttZTgbxB4ag=
> >>> 6aZjPCc9e00raokVPKvx1BxwDOHyCuKNgtBXPMeoHy4=
> >>>
>  243%3A+Make+ProducerConfig+and+ConsumerConfig+constructors+public
> 
> 
>  This is a rather straight forward change, thus I skip the DISCUSS
>  thread and call for a vote immediately.
> 
> 
>  -Matthias
> 
> 
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> >
> >
> >
> > --
> > -- Guozhang
> >
> 
> >>>
> >>
> >>
> >
>
>

[jira] [Created] (KAFKA-6398) Stream-Table join fails, if table is not materialized

2017-12-21 Thread Matthias J. Sax (JIRA)

Matthias J. Sax created KAFKA-6398:
--

 Summary: Stream-Table join fails, if table is not materialized
 Key: KAFKA-6398
 URL: https://issues.apache.org/jira/browse/KAFKA-6398
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 0.11.0.1
Reporter: Matthias J. Sax


Using a non-materialized KTable in a stream-table join fails:

{noformat}
final KTable filteredKTable = builder.table("table-topic").filter(...);
builder.stream("stream-topic").join(filteredKTable,...);
{noformat}

fails with
{noformat}
org.apache.kafka.streams.errors.TopologyBuilderException: Invalid topology 
building: StateStore null is not added yet.

at 
org.apache.kafka.streams.processor.TopologyBuilder.connectProcessorAndStateStore(TopologyBuilder.java:1021)
at 
org.apache.kafka.streams.processor.TopologyBuilder.connectProcessorAndStateStores(TopologyBuilder.java:949)
at 
org.apache.kafka.streams.kstream.internals.KStreamImpl.doStreamTableJoin(KStreamImpl.java:621)
at 
org.apache.kafka.streams.kstream.internals.KStreamImpl.join(KStreamImpl.java:577)
at 
org.apache.kafka.streams.kstream.internals.KStreamImpl.join(KStreamImpl.java:563)
{noformat}

Adding a store name is not sufficient as workaround but fails differently:
{noformat}
final KTable filteredKTable = builder.table("table-topic").filter(..., 
"STORE-NAME");
builder.stream("stream-topic").join(filteredKTable,...);
{noformat}

error:
{noformat}
org.apache.kafka.streams.errors.StreamsException: failed to initialize 
processor KSTREAM-JOIN-05

at 
org.apache.kafka.streams.processor.internals.ProcessorNode.init(ProcessorNode.java:113)
at 
org.apache.kafka.streams.processor.internals.StreamTask.initTopology(StreamTask.java:339)
at 
org.apache.kafka.streams.processor.internals.StreamTask.initialize(StreamTask.java:153)
Caused by: org.apache.kafka.streams.errors.TopologyBuilderException: Invalid 
topology building: Processor KSTREAM-JOIN-05 has no access to 
StateStore KTABLE-SOURCE-STATE-STORE-00
at 
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.getStateStore(ProcessorContextImpl.java:69)
at 
org.apache.kafka.streams.kstream.internals.KTableSourceValueGetterSupplier$KTableSourceValueGetter.init(KTableSourceValueGetterSupplier.java:45)
at 
org.apache.kafka.streams.kstream.internals.KTableFilter$KTableFilterValueGetter.init(KTableFilter.java:121)
at 
org.apache.kafka.streams.kstream.internals.KStreamKTableJoinProcessor.init(KStreamKTableJoinProcessor.java:44)
at 
org.apache.kafka.streams.processor.internals.ProcessorNode$2.run(ProcessorNode.java:53)
at 
org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:201)
at 
org.apache.kafka.streams.processor.internals.ProcessorNode.init(ProcessorNode.java:111)
{noformat}

One can workaround by piping the result through a topic:
{noformat}
final KTable filteredKTable = 
builder.table("table-topic").filter(...).through("TOPIC");;
builder.stream("stream-topic").join(filteredKTable,...);
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

2017-12-21 Thread Randall Hauch

All,

I've updated KIP-145 to reflect my proposal. The proposal addresses SMTs
and a different HeaderConverter default, but I'll be updating my PR (
https://github.com/apache/kafka/pull/4319) soon. Feedback is very welcome!

Best regards,

Randall

On Thu, Dec 14, 2017 at 10:20 AM, Randall Hauch  wrote:

> Hi, Michael. Yeah, I liked your PR a lot, and there definitely are a lot
> of similarities. But here are the more significant differences from my
> perspective (none of which are really that big):
>
> First, your `SubjectConverter` and my `HeaderConverter` are pretty similar
> -- mine is just more closely tied to headers. Also, we used slightly
> different approaches to dealing with the fact that the `Converter`
> interface does not extend `Configurable`, which Connect now uses for
> transforms, connectors, etc. And our implementations take very different
> approaches (see below).
>
> Second, I tried to follow Kafka client's `Header` and `Headers` interfaces
> (at least in concept) so that ConnectRecord has a `Headers` rather than a
> list of headers. It's a minor distinction, but I do think it's important
> for future-proofing to have an interface for the collection to abstract and
> encapsulate logic/behavior as well as leaving room for alternative
> implementations. It also a convenient place to add methods for source
> connectors and SMTs to easily add/modify/remove/transform headers.
>
> Third, our "header converter" implementations are where most of the
> differences lie. Again, this goes back to my assertion that we should make
> the serdes and cast/conversion orthogonal. If we allow sink connectors and
> SMTs to get header values in the type they want (e.g.,
> `Header.valueAsFloat()`), then we can tolerate a bit more variation in how
> the header values are serialized and deserialized, since the serdes
> mechanism doesn't have to get the type exactly right for the sink connector
> and SMT. My `SimpleHeaderConverter` serializes all of the types to strings,
> but during deserialization it attempts to infer the schemas (easy for
> primitive values, a bit harder for structured types). IIUC, neither your
> approach or mine is really able to maintain Struct schemas, but IMO we can
> add that over time with improved/different header converters if people
> really need it.
>
> Fourth, we use different defaults for the serdes implementation. I dislike
> the StringConverter because it converts everything to strings that are then
> difficult to convert back to the original form, especially for the
> structured types. This is why I created the `SimpleHeaderConverter`
> implementation, which doesn't need explicit configuration or explicit
> mapping of header names to types, and thus can be used as the default.
>
> Finally, while I hope that `SimpleHeaderConverter` and its schema
> inference will work most of the time with no special configuration,
> especially since the `Header` interface makes it easy to cast/convert in
> sink connectors and SMTs, I do like how your `PrimativeSubjectConverter`
> allows the user to manually control how the values are serialized. I
> thought of doing something similar, but I think that can be done at a later
> time if/when needed.
>
> I hope that makes sense.
>
> Randall
>
> On Tue, Dec 12, 2017 at 11:35 PM, Michael André Pearce <
> michael.andre.pea...@me.com> wrote:
>
>> Hi Randall
>>
>> What’s the main difference between this and my earlier alternative option
>> PR
>> https://github.com/apache/kafka/pull/2942/files
>>
>> If none then +1.
>> From what I can tell the only difference I make is the headers you
>> support being able to cross convert primitive types eg if value after
>> conversion is integer you can still ask for float and it will type concert
>> if possible.
>>
>> Cheers
>> Mike
>>
>>
>> Sent from my iPhone
>>
>> > On 13 Dec 2017, at 01:36, Randall Hauch  wrote:
>> >
>> > Trying to revive this after several months of inactivity
>> >
>> > I've spent quite a bit of time evaluating the current KIP-145 proposal
>> and
>> > several of the suggested PRs. The original KIP-145 proposal is
>> relatively
>> > minimalist (which is very nice), and it adopts Kafka's approach to
>> headers
>> > where header keys are strings and header values are byte arrays. IMO,
>> this
>> > places too much responsibility on the connector developers to know how
>> to
>> > serialize and deserialize, which means that it's going to be difficult
>> to
>> > assemble into pipelines connectors and stream processors that make
>> > different, incompatible assumptions. It also makes Connect headers very
>> > different than Connect's keys and values, which are generally structured
>> > and describable with Connect schemas. I think we need Connect headers
>> to do
>> > more.
>> >
>> > The other proposals attempt to do more, but even my first proposal
>> doesn't
>> > seem to really provide a solution that works for Connect users and
>> > connector developers. After

Build failed in Jenkins: kafka-trunk-jdk8 #2294

2017-12-21 Thread Apache Jenkins Server

See 


Changes:

[wangguoz] MINOR: Improve Streams Dev Guide content on web docs

--
[...truncated 3.38 MB...]
kafka.utils.CoreUtilsTest > testReadInt STARTED

kafka.utils.CoreUtilsTest > testReadInt PASSED

kafka.utils.CoreUtilsTest > testAtomicGetOrUpdate STARTED

kafka.utils.CoreUtilsTest > testAtomicGetOrUpdate PASSED

kafka.utils.CoreUtilsTest > testUrlSafeBase64EncodeUUID STARTED

kafka.utils.CoreUtilsTest > testUrlSafeBase64EncodeUUID PASSED

kafka.utils.CoreUtilsTest > testCsvMap STARTED

kafka.utils.CoreUtilsTest > testCsvMap PASSED

kafka.utils.CoreUtilsTest > testInLock STARTED

kafka.utils.CoreUtilsTest > testInLock PASSED

kafka.utils.CoreUtilsTest > testTryAll STARTED

kafka.utils.CoreUtilsTest > testTryAll PASSED

kafka.utils.CoreUtilsTest > testSwallow STARTED

kafka.utils.CoreUtilsTest > testSwallow PASSED

kafka.utils.IteratorTemplateTest > testIterator STARTED

kafka.utils.IteratorTemplateTest > testIterator PASSED

kafka.utils.json.JsonValueTest > testJsonObjectIterator STARTED

kafka.utils.json.JsonValueTest > testJsonObjectIterator PASSED

kafka.utils.json.JsonValueTest > testDecodeLong STARTED

kafka.utils.json.JsonValueTest > testDecodeLong PASSED

kafka.utils.json.JsonValueTest > testAsJsonObject STARTED

kafka.utils.json.JsonValueTest > testAsJsonObject PASSED

kafka.utils.json.JsonValueTest > testDecodeDouble STARTED

kafka.utils.json.JsonValueTest > testDecodeDouble PASSED

kafka.utils.json.JsonValueTest > testDecodeOption STARTED

kafka.utils.json.JsonValueTest > testDecodeOption PASSED

kafka.utils.json.JsonValueTest > testDecodeString STARTED

kafka.utils.json.JsonValueTest > testDecodeString PASSED

kafka.utils.json.JsonValueTest > testJsonValueToString STARTED

kafka.utils.json.JsonValueTest > testJsonValueToString PASSED

kafka.utils.json.JsonValueTest > testAsJsonObjectOption STARTED

kafka.utils.json.JsonValueTest > testAsJsonObjectOption PASSED

kafka.utils.json.JsonValueTest > testAsJsonArrayOption STARTED

kafka.utils.json.JsonValueTest > testAsJsonArrayOption PASSED

kafka.utils.json.JsonValueTest > testAsJsonArray STARTED

kafka.utils.json.JsonValueTest > testAsJsonArray PASSED

kafka.utils.json.JsonValueTest > testJsonValueHashCode STARTED

kafka.utils.json.JsonValueTest > testJsonValueHashCode PASSED

kafka.utils.json.JsonValueTest > testDecodeInt STARTED

kafka.utils.json.JsonValueTest > testDecodeInt PASSED

kafka.utils.json.JsonValueTest > testDecodeMap STARTED

kafka.utils.json.JsonValueTest > testDecodeMap PASSED

kafka.utils.json.JsonValueTest > testDecodeSeq STARTED

kafka.utils.json.JsonValueTest > testDecodeSeq PASSED

kafka.utils.json.JsonValueTest > testJsonObjectGet STARTED

kafka.utils.json.JsonValueTest > testJsonObjectGet PASSED

kafka.utils.json.JsonValueTest > testJsonValueEquals STARTED

kafka.utils.json.JsonValueTest > testJsonValueEquals PASSED

kafka.utils.json.JsonValueTest > testJsonArrayIterator STARTED

kafka.utils.json.JsonValueTest > testJsonArrayIterator PASSED

kafka.utils.json.JsonValueTest > testJsonObjectApply STARTED

kafka.utils.json.JsonValueTest > testJsonObjectApply PASSED

kafka.utils.json.JsonValueTest > testDecodeBoolean STARTED

kafka.utils.json.JsonValueTest > testDecodeBoolean PASSED

kafka.producer.AsyncProducerTest > testFailedSendRetryLogic STARTED

kafka.producer.AsyncProducerTest > testFailedSendRetryLogic PASSED

kafka.producer.AsyncProducerTest > testQueueTimeExpired STARTED

kafka.producer.AsyncProducerTest > testQueueTimeExpired PASSED

kafka.producer.AsyncProducerTest > testPartitionAndCollateEvents STARTED

kafka.producer.AsyncProducerTest > testPartitionAndCollateEvents PASSED

kafka.producer.AsyncProducerTest > testBatchSize STARTED

kafka.producer.AsyncProducerTest > testBatchSize PASSED

kafka.producer.AsyncProducerTest > testSerializeEvents STARTED

kafka.producer.AsyncProducerTest > testSerializeEvents PASSED

kafka.producer.AsyncProducerTest > testProducerQueueSize STARTED

kafka.producer.AsyncProducerTest > testProducerQueueSize PASSED

kafka.producer.AsyncProducerTest > testRandomPartitioner STARTED

kafka.producer.AsyncProducerTest > testRandomPartitioner PASSED

kafka.producer.AsyncProducerTest > testInvalidConfiguration STARTED

kafka.producer.AsyncProducerTest > testInvalidConfiguration PASSED

kafka.producer.AsyncProducerTest > testInvalidPartition STARTED

kafka.producer.AsyncProducerTest > testInvalidPartition PASSED

kafka.producer.AsyncProducerTest > testNoBroker STARTED

kafka.producer.AsyncProducerTest > testNoBroker PASSED

kafka.producer.AsyncProducerTest > testProduceAfterClosed STARTED

kafka.producer.AsyncProducerTest > testProduceAfterClosed PASSED

kafka.producer.AsyncProducerTest > testJavaProducer STARTED

kafka.producer.AsyncProducerTest > testJavaProducer PASSED

kafka.producer.AsyncProducerTest > testIncompatibleEncoder STARTED

Re: [VOTE] KIP-243: Make ProducerConfig and ConsumerConfig constructors public

2017-12-21 Thread Matthias J. Sax

I personally love the builder pattern idea. There was some push back in
the past though from some people.

cf https://issues.apache.org/jira/browse/KAFKA-4436

Happy to propose the builder pattern but than we should have a proper
DISCUSS thread. Maybe we do this as a follow up and just do this KIP as-is?


-Matthias

On 12/21/17 10:28 AM, Jason Gustafson wrote:
> Hey Matthias,
> 
> Let me suggest an alternative. As you have mentioned, these config classes
> do not give users much benefit currently. Maybe we change that? I think
> many users would appreciate having a builder for configuration since it
> provides type safety and is generally a much friendlier pattern to work
> with programmatically. Users could then do something like this:
> 
> ConsumerConfig config = ConsumerConfig.newBuilder()
> .setBootstrapServers("localhost:9092")
> .setGroupId("group")
> .setRequestTimeout(15, TimeUnit.SECONDS)
> .build();
> 
> Consumer consumer = new KafkaConsumer(config);
> 
> An additional benefit of this is that it gives us a better way to expose
> config deprecations. In any case, it would make it less odd to expose the
> public constructor without giving users anything useful to do with the
> class.
> 
> What do you think?
> 
> -Jason
> 
> On Wed, Dec 20, 2017 at 5:59 PM, Matthias J. Sax 
> wrote:
> 
>> It's tailored for internal usage. I think client constructors don't
>> benefit from accepting those config objects. We just want to be able to
>> access the default values for certain parameters.
>>
>> From a user point of view, it's actually boiler plate code if you pass
>> in a config object instead of a plain Properties object because the
>> config object itself is immutable.
>>
>> I actually create a JIRA to remove the constructors from KafkaStreams
>> that do accept StreamsConfig for exact this reason:
>> https://issues.apache.org/jira/browse/KAFKA-6386
>>
>>
>> -Matthias
>>
>>
>> On 12/20/17 3:33 PM, Jason Gustafson wrote:
>>> Hi Matthias,
>>>
>>> Isn't it a little weird to make these constructors public but not also
>>> expose the corresponding client constructors that use them?
>>>
>>> -Jason
>>>
>>> On Tue, Dec 19, 2017 at 9:30 AM, Bill Bejeck  wrote:
>>>
 +1

 On Tue, Dec 19, 2017 at 12:09 PM, Guozhang Wang 
 wrote:

> +1
>
> On Tue, Dec 19, 2017 at 1:49 AM, Tom Bentley 
> wrote:
>
>> +1
>>
>> On 18 December 2017 at 23:28, Vahid S Hashemian <
> vahidhashem...@us.ibm.com
>>>
>> wrote:
>>
>>> +1
>>>
>>> Thanks for the KIP.
>>>
>>> --Vahid
>>>
>>>
>>>
>>> From:   Ted Yu 
>>> To: dev@kafka.apache.org
>>> Date:   12/18/2017 02:45 PM
>>> Subject:Re: [VOTE] KIP-243: Make ProducerConfig and
>> ConsumerConfig
>>> constructors public
>>>
>>>
>>>
>>> +1
>>>
>>> nit: via "copy and past" an 'e' is missing at the end.
>>>
>>> On Mon, Dec 18, 2017 at 2:38 PM, Matthias J. Sax <
> matth...@confluent.io>
>>> wrote:
>>>
 Hi,

 I want to propose the following KIP:

>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
>>> apache.org_confluence_display_KAFKA_KIP-2D=DwIBaQ=jf_
>>> iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
>>> kjJc7uSVcviKUc=JToRX4-HeVsRoOekIz18ht-YLMe-T21MttZTgbxB4ag=
>>> 6aZjPCc9e00raokVPKvx1BxwDOHyCuKNgtBXPMeoHy4=
>>>
 243%3A+Make+ProducerConfig+and+ConsumerConfig+constructors+public


 This is a rather straight forward change, thus I skip the DISCUSS
 thread and call for a vote immediately.


 -Matthias


>>>
>>>
>>>
>>>
>>>
>>
>
>
>
> --
> -- Guozhang
>

>>>
>>
>>
> 



signature.asc
Description: OpenPGP digital signature

[GitHub] kafka pull request #4252: Migrate Streams Dev Guide content to AK

2017-12-21 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4252


---

[jira] [Created] (KAFKA-6397) Consumer should not block setting initial positions of unavailable partitions

2017-12-21 Thread Jason Gustafson (JIRA)

Jason Gustafson created KAFKA-6397:
--

 Summary: Consumer should not block setting initial positions of 
unavailable partitions
 Key: KAFKA-6397
 URL: https://issues.apache.org/jira/browse/KAFKA-6397
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson
Assignee: Jason Gustafson


Currently the consumer will block in poll() after receiving its assignment in 
order to set the starting offset for every assigned partition. If the topic is 
deleted or if a partition is unavailable, the consumer can be stuck 
indefinitely. Most of the time this is not a problem since the starting offset 
is obtained from the committed offsets, which does not depend on partition 
availability. However, if there are no committed offsets or if the user has 
manually called {{seekToBeginning}} or {{seekToEnd}}, then we will need to do a 
lookup for the starting offset from the partition leader, which will stall the 
consumer until the partition is available or recreated. It would be better to 
let the consumer fetch on partitions which are available and periodically check 
availability for the rest. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] KIP-243: Make ProducerConfig and ConsumerConfig constructors public

2017-12-21 Thread Jason Gustafson

Hey Matthias,

Let me suggest an alternative. As you have mentioned, these config classes
do not give users much benefit currently. Maybe we change that? I think
many users would appreciate having a builder for configuration since it
provides type safety and is generally a much friendlier pattern to work
with programmatically. Users could then do something like this:

ConsumerConfig config = ConsumerConfig.newBuilder()
.setBootstrapServers("localhost:9092")
.setGroupId("group")
.setRequestTimeout(15, TimeUnit.SECONDS)
.build();

Consumer consumer = new KafkaConsumer(config);

An additional benefit of this is that it gives us a better way to expose
config deprecations. In any case, it would make it less odd to expose the
public constructor without giving users anything useful to do with the
class.

What do you think?

-Jason

On Wed, Dec 20, 2017 at 5:59 PM, Matthias J. Sax 
wrote:

> It's tailored for internal usage. I think client constructors don't
> benefit from accepting those config objects. We just want to be able to
> access the default values for certain parameters.
>
> From a user point of view, it's actually boiler plate code if you pass
> in a config object instead of a plain Properties object because the
> config object itself is immutable.
>
> I actually create a JIRA to remove the constructors from KafkaStreams
> that do accept StreamsConfig for exact this reason:
> https://issues.apache.org/jira/browse/KAFKA-6386
>
>
> -Matthias
>
>
> On 12/20/17 3:33 PM, Jason Gustafson wrote:
> > Hi Matthias,
> >
> > Isn't it a little weird to make these constructors public but not also
> > expose the corresponding client constructors that use them?
> >
> > -Jason
> >
> > On Tue, Dec 19, 2017 at 9:30 AM, Bill Bejeck  wrote:
> >
> >> +1
> >>
> >> On Tue, Dec 19, 2017 at 12:09 PM, Guozhang Wang 
> >> wrote:
> >>
> >>> +1
> >>>
> >>> On Tue, Dec 19, 2017 at 1:49 AM, Tom Bentley 
> >>> wrote:
> >>>
>  +1
> 
>  On 18 December 2017 at 23:28, Vahid S Hashemian <
> >>> vahidhashem...@us.ibm.com
> >
>  wrote:
> 
> > +1
> >
> > Thanks for the KIP.
> >
> > --Vahid
> >
> >
> >
> > From:   Ted Yu 
> > To: dev@kafka.apache.org
> > Date:   12/18/2017 02:45 PM
> > Subject:Re: [VOTE] KIP-243: Make ProducerConfig and
>  ConsumerConfig
> > constructors public
> >
> >
> >
> > +1
> >
> > nit: via "copy and past" an 'e' is missing at the end.
> >
> > On Mon, Dec 18, 2017 at 2:38 PM, Matthias J. Sax <
> >>> matth...@confluent.io>
> > wrote:
> >
> >> Hi,
> >>
> >> I want to propose the following KIP:
> >>
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
> > apache.org_confluence_display_KAFKA_KIP-2D=DwIBaQ=jf_
> > iaSHvJObTbx-siA1ZOg=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
> > kjJc7uSVcviKUc=JToRX4-HeVsRoOekIz18ht-YLMe-T21MttZTgbxB4ag=
> > 6aZjPCc9e00raokVPKvx1BxwDOHyCuKNgtBXPMeoHy4=
> >
> >> 243%3A+Make+ProducerConfig+and+ConsumerConfig+constructors+public
> >>
> >>
> >> This is a rather straight forward change, thus I skip the DISCUSS
> >> thread and call for a vote immediately.
> >>
> >>
> >> -Matthias
> >>
> >>
> >
> >
> >
> >
> >
> 
> >>>
> >>>
> >>>
> >>> --
> >>> -- Guozhang
> >>>
> >>
> >
>
>

[GitHub] kafka pull request #4350: Cached hashCode of a Node instance since it is imm...

2017-12-21 Thread esevastyanov

GitHub user esevastyanov opened a pull request:

https://github.com/apache/kafka/pull/4350

Cached hashCode of a Node instance since it is immutable

`Node` structure is immutable so it is possible to cache `hashCode` of a 
`Node` instance as it's done in the `TopicPartition` class.
Faced with the performance degradation in case of high load and large 
number of brokers (100), topics (150) and partitions (350). Made several 
diagnostic records with the java flight recorder and found that the method 
`HashSet::contains` in 
[`RecordAccumulator::ready`](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java#L423)
 takes about 40% of the whole time of the application. It is caused by 
re-calculating a hash code of a 
[leader](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java#L433)
 (`Node` instance) for every [batch 
entry](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java#L429).
 The cached hash code solved this issue and the corresponding time of 
`HashSet::contains` in `RecordAccumulator::ready` decreased to ~2%.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/esevastyanov/kafka node-hash

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4350.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4350


commit 736eb4c1bfed561cdd13cfc384ef1c6838faed59
Author: esevastyanov 
Date:   2017-12-21T15:19:08Z

Cached hashCode of a Node instance since it is immutable.




---

Re: KIP-244: Add Record Header support to Kafka Streams

2017-12-21 Thread Bill Bejeck

Jorge,

Thanks for the KIP, I know this is a feature others in the community have
been interested in getting into Kafka Streams.

I took a quick pass over it, and I have one initial question.

We recently reduced overloads with KIP-182, and in this KIP we are
increasing them again.

I can see from the KIP why they are necessary, but I'm wondering if there
is something else we can do to cut down on the overloads introduced.  I
don't have any sound suggestions ATM, so I'll have to think about it some
more, but I wanted to put the thought out there.

Thanks,
Bill

On Thu, Dec 21, 2017 at 9:06 AM, Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:

> Hi all,
>
> I have created a KIP to add Record Headers support to Kafka Streams API:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 244%3A+Add+Record+Header+support+to+Kafka+Streams
>
>
> The main goal is to be able to use headers to filter, map and process
> records as streams. Stateful processing (joins, windows) are not
> considered.
>
> Proposed changes/Draft:
> https://github.com/apache/kafka/compare/trunk...jeqo:streams-headers
>
> Feedback and suggestions are more than welcome.
>
> Cheers,
>
> Jorge.
>

[jira] [Created] (KAFKA-6396) Possibly kafka-connect converter should be able to stop processing chain

2017-12-21 Thread Alexander Koval (JIRA)

Alexander Koval created KAFKA-6396:
--

 Summary: Possibly kafka-connect converter should be able to stop 
processing chain
 Key: KAFKA-6396
 URL: https://issues.apache.org/jira/browse/KAFKA-6396
 Project: Kafka
  Issue Type: Wish
  Components: KafkaConnect
Affects Versions: 1.0.0
Reporter: Alexander Koval
Priority: Minor


At present only transformations can discard records returning null. But I think 
sometimes it would be nice to discard processing chain after converting 
message. For example I have some tags shipped with a message key and I want to 
stop processing the message after converting its key (there are a lot of 
messages and I don't want to deserialize message values that I don't need).

At the moment to do that I should disable converters and move message 
deserializing to the transformation chain:

{code}
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter

transforms=proto,catalog
transforms.proto.type=company.evo.kafka.ProtobufTransformation
transforms.proto.key.protobuf.class=company.evo.uaprom.indexator.KeyProto$KeyMessage
transforms.proto.value.protobuf.class=company.evo.uaprom.indexator.catalog.CompanyProto$UniversalCompanyMessage
transforms.proto.tag=catalog
{code}

If 
[WorkerSinkTask|https://github.com/apache/kafka/blob/1.0.0/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L453]
 checked converted values on {{null}} it would solved my problem more gracefully




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

KIP-244: Add Record Header support to Kafka Streams

2017-12-21 Thread Jorge Esteban Quilcate Otoya

Hi all,

I have created a KIP to add Record Headers support to Kafka Streams API:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-244%3A+Add+Record+Header+support+to+Kafka+Streams


The main goal is to be able to use headers to filter, map and process
records as streams. Stateful processing (joins, windows) are not
considered.

Proposed changes/Draft:
https://github.com/apache/kafka/compare/trunk...jeqo:streams-headers

Feedback and suggestions are more than welcome.

Cheers,

Jorge.

[jira] [Created] (KAFKA-6395) KIP: Add Record Header support to Kafka Streams

2017-12-21 Thread Jorge Quilcate (JIRA)

Jorge Quilcate created KAFKA-6395:
-

 Summary: KIP: Add Record Header support to Kafka Streams 
 Key: KAFKA-6395
 URL: https://issues.apache.org/jira/browse/KAFKA-6395
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Jorge Quilcate
Assignee: Jorge Quilcate


KIP documentation: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-244%3A+Add+Record+Header+support+to+Kafka+Streams



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] KIP-239 Add queryableStoreName() to GlobalKTable

2017-12-21 Thread Damian Guy

+1

On Wed, 20 Dec 2017 at 21:09 Ted Yu  wrote:

> Ping for more (binding) votes.
>
> The pull request is ready.
>
> On Fri, Dec 15, 2017 at 12:57 PM, Guozhang Wang 
> wrote:
>
> > +1 (binding), thanks!
> >
> > On Fri, Dec 15, 2017 at 11:56 AM, Ted Yu  wrote:
> >
> > > Hi,
> > > Here is the discussion thread:
> > >
> > > http://search-hadoop.com/m/Kafka/uyzND12QnH514pPO9?subj=
> > > Re+DISCUSS+KIP+239+Add+queryableStoreName+to+GlobalKTable
> > >
> > > Please vote on this KIP.
> > >
> > > Thanks
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>

Re: [VOTE] Allowing write access to GitHub repositories (aka GitBox)

[jira] [Resolved] (KAFKA-3496) Add reconnect attemps policies for client

[jira] [Resolved] (KAFKA-5895) Gradle 3.0+ is needed on the build

Re: [VOTE] Allowing write access to GitHub repositories (aka GitBox)

Re: [VOTE] Allowing write access to GitHub repositories (aka GitBox)

Re: [VOTE] KIP-237: More Controller Health Metrics

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

[GitHub] kafka pull request #4351: kafka-6320: move ZK metrics in KafkaHealthCheck to...

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Re: [VOTE] KIP-243: Make ProducerConfig and ConsumerConfig constructors public

[jira] [Created] (KAFKA-6398) Stream-Table join fails, if table is not materialized

Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

Build failed in Jenkins: kafka-trunk-jdk8 #2294

Re: [VOTE] KIP-243: Make ProducerConfig and ConsumerConfig constructors public

[GitHub] kafka pull request #4252: Migrate Streams Dev Guide content to AK

[jira] [Created] (KAFKA-6397) Consumer should not block setting initial positions of unavailable partitions

Re: [VOTE] KIP-243: Make ProducerConfig and ConsumerConfig constructors public

[GitHub] kafka pull request #4350: Cached hashCode of a Node instance since it is imm...

Re: KIP-244: Add Record Header support to Kafka Streams

[jira] [Created] (KAFKA-6396) Possibly kafka-connect converter should be able to stop processing chain

KIP-244: Add Record Header support to Kafka Streams

[jira] [Created] (KAFKA-6395) KIP: Add Record Header support to Kafka Streams

Re: [VOTE] KIP-239 Add queryableStoreName() to GlobalKTable

23 matches

Site Navigation

Mail list logo

Footer information