date:20180626

I started to read this KIP which contains a lot of materials.

One suggestion:

.suppress(
new Suppression()


Do you think it would be more consistent with the rest of Streams data
structures by supporting `of` ?

Suppression.of(Duration.ofMinutes(10))


Cheers



On Tue, Jun 26, 2018 at 1:11 PM, John Roesler  wrote:

> Hello devs and users,
>
> Please take some time to consider this proposal for Kafka Streams:
>
> KIP-328: Ability to suppress updates for KTables
>
> link: https://cwiki.apache.org/confluence/x/sQU0BQ
>
> The basic idea is to provide:
> * more usable control over update rate (vs the current state store caches)
> * the final-result-for-windowed-computations feature which several people
> have requested
>
> I look forward to your feedback!
>
> Thanks,
> -John
>

Requesting Permission To Create KIP And Assign JIRAs

2018-06-26 Thread Kevin Lu

Hi All,

I would like to start contributing to Kafka but I do not have access to
create KIPs or assign JIRA to myself.

Can someone set it up for me?

Confluence id: lu.kevin
Jira username: lu.kevin

Email: lu.ke...@berkeley.edu

Thanks!

Regards,
Kevin

Jenkins build is back to normal : kafka-trunk-jdk8 #2772

See

Re: [DISCUSS] KIP-325: Extend Consumer Group Command to Show Beginning Offsets

Thank you!

On Tue, Jun 26, 2018 at 8:47 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Thanks for the feedback. The KIP is updated to also include a "partition
> size" column.
>
> --Vahid
>
>
>
>
> From:   Ted Yu 
> To: dev@kafka.apache.org
> Date:   06/26/2018 06:21 PM
> Subject:Re: [DISCUSS] KIP-325: Extend Consumer Group Command to
> Show Beginning Offsets
>
>
>
> nit:
>
> bq. leaving this empty for compacted topics
>
> Some user(s) may be confused by empty partition size. How about emitting
> 'compacted' for compacted topics ?
>
> Cheers
>
> On Tue, Jun 26, 2018 at 4:42 PM, Gwen Shapira  wrote:
>
> > It will be. In my experience most topics aren't compacted, so it will
> still
> > be valuable. If not difficult, leaving this empty for compacted topics
> to
> > avoid confusion will also be nice.
> >
> > On Tue, Jun 26, 2018 at 4:29 PM, Vahid S Hashemian <
> > vahidhashem...@us.ibm.com> wrote:
> >
> > > Hi Gwen,
> > >
> > > Thanks for the feedback.
> > > Regarding the partition size, couldn't "end offset - start offset" be
> > > misleading for compacted topics?
> > >
> > > --Vahid
> > >
> > >
> > >
> > >
> > > From:   Gwen Shapira 
> > > To: dev 
> > > Date:   06/26/2018 02:36 PM
> > > Subject:Re: [DISCUSS] KIP-325: Extend Consumer Group Command
> to
> > > Show Beginning Offsets
> > >
> > >
> > >
> > > Small suggestion: you can also add a "partition size" column -
> difference
> > > between log-end and log-start. We've had users ask for this.
> > >
> > > On Tue, Jun 26, 2018 at 2:34 PM, Gwen Shapira 
> wrote:
> > >
> > > > This will be useful! Thank you :)
> > > >
> > > > On Tue, Jun 26, 2018 at 11:23 AM, Vahid S Hashemian <
> > > > vahidhashem...@us.ibm.com> wrote:
> > > >
> > > >> Hi everyone,
> > > >>
> > > >> I have created a trivial KIP to improve the offset reporting of the
> > > >> consumer group command:
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-325%
>
> > >
> > > >> 3A+Extend+Consumer+Group+Command+to+Show+Beginning+Offsets
> > > >> Looking forward to your feedback!
> > > >>
> > > >> Thanks.
> > > >> --Vahid
> > > >>
> > > >>
> > > >>
> > > >
> > > >
> > > > --
> > > > *Gwen Shapira*
> > > > Product Manager | Confluent
> > > > 650.450.2760 | @gwenshap
> > > > Follow us: Twitter <
> > >
> https://twitter.com/ConfluentInc
>
> > > > | blog
> > > > <
> > >
> http://www.confluent.io/blog
>
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > *Gwen Shapira*
> > > Product Manager | Confluent
> > > 650.450.2760 | @gwenshap
> > > Follow us: Twitter <
> > >
> https://twitter.com/ConfluentInc
>
> > > > | blog
> > > <
> > >
> http://www.confluent.io/blog
>
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> > --
> > *Gwen Shapira*
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter <
> https://twitter.com/ConfluentInc
> > | blog
> > <
> http://www.confluent.io/blog
> >
> >
>
>
>
>
>


-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter  | blog

Build failed in Jenkins: kafka-2.0-jdk8 #58

See 


Changes:

[wangguoz] HOTFIX: update Streams security docs

--
[...truncated 430.29 KB...]

kafka.controller.ReplicaStateMachineTest > 
testInvalidOfflineReplicaToReplicaDeletionSuccessfulTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOfflineReplicaToReplicaDeletionSuccessfulTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testOfflineReplicaToOnlineReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testOfflineReplicaToOnlineReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionStartedToNonexistentReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionStartedToNonexistentReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testOfflineReplicaToReplicaDeletionStartedTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testOfflineReplicaToReplicaDeletionStartedTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionSuccessfulToNewReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionSuccessfulToNewReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionIneligibleToNonexistentReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionIneligibleToNonexistentReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testReplicaDeletionSuccessfulToNonexistentReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testReplicaDeletionSuccessfulToNonexistentReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOnlineReplicaToNonexistentReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOnlineReplicaToNonexistentReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOfflineReplicaToReplicaDeletionIneligibleTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOfflineReplicaToReplicaDeletionIneligibleTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testNewReplicaToOfflineReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testNewReplicaToOfflineReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionSuccessfulToOnlineReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionSuccessfulToOnlineReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNonexistentReplicaToReplicaDeletionStartedTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNonexistentReplicaToReplicaDeletionStartedTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOnlineReplicaToReplicaDeletionSuccessfulTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOnlineReplicaToReplicaDeletionSuccessfulTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOfflineReplicaToNonexistentReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOfflineReplicaToNonexistentReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOnlineReplicaToReplicaDeletionIneligibleTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidOnlineReplicaToReplicaDeletionIneligibleTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionSuccessfulToReplicaDeletionStartedTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionSuccessfulToReplicaDeletionStartedTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNewReplicaToReplicaDeletionSuccessfulTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNewReplicaToReplicaDeletionSuccessfulTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionIneligibleToReplicaDeletionStartedTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionIneligibleToReplicaDeletionStartedTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionStartedToOfflineReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidReplicaDeletionStartedToOfflineReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNewReplicaToReplicaDeletionStartedTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNewReplicaToReplicaDeletionStartedTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testOnlineReplicaToOnlineReplicaTransition STARTED

kafka.controller.ReplicaStateMachineTest > 
testOnlineReplicaToOnlineReplicaTransition PASSED

kafka.controller.ReplicaStateMachineTest > 
testInvalidNewReplicaToReplicaDeletionIneligibleTransition

Re: [DISCUSS] KIP-325: Extend Consumer Group Command to Show Beginning Offsets

2018-06-26 Thread Vahid S Hashemian

Thanks for the feedback. The KIP is updated to also include a "partition 
size" column.

--Vahid




From:   Ted Yu 
To: dev@kafka.apache.org
Date:   06/26/2018 06:21 PM
Subject:Re: [DISCUSS] KIP-325: Extend Consumer Group Command to 
Show Beginning Offsets



nit:

bq. leaving this empty for compacted topics

Some user(s) may be confused by empty partition size. How about emitting
'compacted' for compacted topics ?

Cheers

On Tue, Jun 26, 2018 at 4:42 PM, Gwen Shapira  wrote:

> It will be. In my experience most topics aren't compacted, so it will 
still
> be valuable. If not difficult, leaving this empty for compacted topics 
to
> avoid confusion will also be nice.
>
> On Tue, Jun 26, 2018 at 4:29 PM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
> > Hi Gwen,
> >
> > Thanks for the feedback.
> > Regarding the partition size, couldn't "end offset - start offset" be
> > misleading for compacted topics?
> >
> > --Vahid
> >
> >
> >
> >
> > From:   Gwen Shapira 
> > To: dev 
> > Date:   06/26/2018 02:36 PM
> > Subject:Re: [DISCUSS] KIP-325: Extend Consumer Group Command 
to
> > Show Beginning Offsets
> >
> >
> >
> > Small suggestion: you can also add a "partition size" column - 
difference
> > between log-end and log-start. We've had users ask for this.
> >
> > On Tue, Jun 26, 2018 at 2:34 PM, Gwen Shapira  
wrote:
> >
> > > This will be useful! Thank you :)
> > >
> > > On Tue, Jun 26, 2018 at 11:23 AM, Vahid S Hashemian <
> > > vahidhashem...@us.ibm.com> wrote:
> > >
> > >> Hi everyone,
> > >>
> > >> I have created a trivial KIP to improve the offset reporting of the
> > >> consumer group command:
> > >>
> > 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-325%

> >
> > >> 3A+Extend+Consumer+Group+Command+to+Show+Beginning+Offsets
> > >> Looking forward to your feedback!
> > >>
> > >> Thanks.
> > >> --Vahid
> > >>
> > >>
> > >>
> > >
> > >
> > > --
> > > *Gwen Shapira*
> > > Product Manager | Confluent
> > > 650.450.2760 | @gwenshap
> > > Follow us: Twitter <
> > 
https://twitter.com/ConfluentInc

> > > | blog
> > > <
> > 
http://www.confluent.io/blog

> > >
> > >
> > >
> >
> >
> > --
> > *Gwen Shapira*
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter <
> > 
https://twitter.com/ConfluentInc

> > > | blog
> > <
> > 
http://www.confluent.io/blog

> > >
> >
> >
> >
> >
> >
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter <
https://twitter.com/ConfluentInc
> | blog
> <
http://www.confluent.io/blog
>
>

Build failed in Jenkins: kafka-trunk-jdk10 #255

See 


Changes:

[wangguoz] HOTFIX: update Streams security docs

--
[...truncated 1.53 MB...]

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithSuccessOnAddPartitionsWhenStateIsCompleteAbort PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldWaitForCommitToCompleteOnHandleInitPidAndExistingTransactionInPrepareAbortState
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldWaitForCommitToCompleteOnHandleInitPidAndExistingTransactionInPrepareAbortState
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithErrorsNoneOnAddPartitionWhenNoErrorsAndPartitionsTheSame 
STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithErrorsNoneOnAddPartitionWhenNoErrorsAndPartitionsTheSame PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithInvalidRequestOnEndTxnWhenTransactionalIdIsEmpty STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithInvalidRequestOnEndTxnWhenTransactionalIdIsEmpty PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithInvalidRequestAddPartitionsToTransactionWhenTransactionalIdIsEmpty
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithInvalidRequestAddPartitionsToTransactionWhenTransactionalIdIsEmpty
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAppendPrepareAbortToLogOnEndTxnWhenStatusIsOngoingAndResultIsAbort STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAppendPrepareAbortToLogOnEndTxnWhenStatusIsOngoingAndResultIsAbort PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAcceptInitPidAndReturnNextPidWhenTransactionalIdIsNull STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAcceptInitPidAndReturnNextPidWhenTransactionalIdIsNull PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRemoveTransactionsForPartitionOnEmigration STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRemoveTransactionsForPartitionOnEmigration PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldWaitForCommitToCompleteOnHandleInitPidAndExistingTransactionInPrepareCommitState
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldWaitForCommitToCompleteOnHandleInitPidAndExistingTransactionInPrepareCommitState
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAbortExpiredTransactionsInOngoingStateAndBumpEpoch STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAbortExpiredTransactionsInOngoingStateAndBumpEpoch PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldReturnInvalidTxnRequestOnEndTxnRequestWhenStatusIsCompleteCommitAndResultIsNotCommit
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldReturnInvalidTxnRequestOnEndTxnRequestWhenStatusIsCompleteCommitAndResultIsNotCommit
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldReturnOkOnEndTxnWhenStatusIsCompleteCommitAndResultIsCommit STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldReturnOkOnEndTxnWhenStatusIsCompleteCommitAndResultIsCommit PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionsOnAddPartitionsWhenStateIsPrepareCommit 
STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionsOnAddPartitionsWhenStateIsPrepareCommit 
PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldIncrementEpochAndUpdateMetadataOnHandleInitPidWhenExistingCompleteTransaction
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldIncrementEpochAndUpdateMetadataOnHandleInitPidWhenExistingCompleteTransaction
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldGenerateNewProducerIdIfEpochsExhausted STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldGenerateNewProducerIdIfEpochsExhausted PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithNotCoordinatorOnInitPidWhenNotCoordinator STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithNotCoordinatorOnInitPidWhenNotCoordinator PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionOnAddPartitionsWhenStateIsPrepareAbort 
STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionOnAddPartitionsWhenStateIsPrepareAbort 
PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldInitPidWithEpochZeroForNewTransactionalId STARTED

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

2018-06-26 Thread Dong Lin

Hey Jason,

Thanks for the explanation.

Please correct me if this is wrong. The "unknown truncation offset"
scenario happens when consumer does not have the full leaderEpoch -> offset
mapping. In this case we can still use the KIP-101-based approach to
truncate offset to "start offset of the first Leader Epoch larger than last
epoch of the consumer" but it may be inaccurate. So the KIP chooses to use
the timestamp-based approach which is also best-effort.

If this understanding is correct, for "closest" offset reset policy and
"unknown truncation offset" scenario, I am wondering whether it maybe
better to replace timestamp-based approach with KIP-101 based approach. In
comparison to timestamp-based approach, the KIP-101-based approach seems to
simplify the API a bit since user does not need to understand timestamp.
Similar to the timestamp-based approach, both approaches are best-effort
and do not guarantee that consumer can consume all messages. It is not like
KIP-279 which guarantees that follower broker can consume all messages from
the leader.

Then it seems that the remaining difference is mostly about accuracy, i.e.
how much message will be duplicated or missed in the "unknown truncation
offset" scenario. Not sure either one is clearly better than the other.
Note that there are two scenarios mentioned in KIP-279 which are not
addressed by KIP-101. Both scenarios require quick leadership change
between brokers, which seems to suggest that the offset based obtained
by "start
offset of the first Leader Epoch larger than last epoch of the consumer"
under these two scenarios may be very close to the offset obtained by the
message timestamp. Does this sound reasonable?

Good point that users on v1 format can get benefit with timestamp based
approach. On the other hand it seems like a short term benefit for users
who have not migrated. I am just not sure whether it is more important than
designing a better API.

Also, for both "latest" and "earliest" reset policy, do you think it would
make sense to also use the KIP-101 based approach to truncate offset for
the "unknown truncation offset" scenario?


Thanks,
Dong

Re: [DISCUSS] KIP-325: Extend Consumer Group Command to Show Beginning Offsets

nit:

bq. leaving this empty for compacted topics

Some user(s) may be confused by empty partition size. How about emitting
'compacted' for compacted topics ?

Cheers

On Tue, Jun 26, 2018 at 4:42 PM, Gwen Shapira  wrote:

> It will be. In my experience most topics aren't compacted, so it will still
> be valuable. If not difficult, leaving this empty for compacted topics to
> avoid confusion will also be nice.
>
> On Tue, Jun 26, 2018 at 4:29 PM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
> > Hi Gwen,
> >
> > Thanks for the feedback.
> > Regarding the partition size, couldn't "end offset - start offset" be
> > misleading for compacted topics?
> >
> > --Vahid
> >
> >
> >
> >
> > From:   Gwen Shapira 
> > To: dev 
> > Date:   06/26/2018 02:36 PM
> > Subject:Re: [DISCUSS] KIP-325: Extend Consumer Group Command to
> > Show Beginning Offsets
> >
> >
> >
> > Small suggestion: you can also add a "partition size" column - difference
> > between log-end and log-start. We've had users ask for this.
> >
> > On Tue, Jun 26, 2018 at 2:34 PM, Gwen Shapira  wrote:
> >
> > > This will be useful! Thank you :)
> > >
> > > On Tue, Jun 26, 2018 at 11:23 AM, Vahid S Hashemian <
> > > vahidhashem...@us.ibm.com> wrote:
> > >
> > >> Hi everyone,
> > >>
> > >> I have created a trivial KIP to improve the offset reporting of the
> > >> consumer group command:
> > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-325%
> >
> > >> 3A+Extend+Consumer+Group+Command+to+Show+Beginning+Offsets
> > >> Looking forward to your feedback!
> > >>
> > >> Thanks.
> > >> --Vahid
> > >>
> > >>
> > >>
> > >
> > >
> > > --
> > > *Gwen Shapira*
> > > Product Manager | Confluent
> > > 650.450.2760 | @gwenshap
> > > Follow us: Twitter <
> > https://twitter.com/ConfluentInc
> > > | blog
> > > <
> > http://www.confluent.io/blog
> > >
> > >
> > >
> >
> >
> > --
> > *Gwen Shapira*
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter <
> > https://twitter.com/ConfluentInc
> > > | blog
> > <
> > http://www.confluent.io/blog
> > >
> >
> >
> >
> >
> >
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter  | blog
> 
>

Re: [DISCUSS] KIP-325: Extend Consumer Group Command to Show Beginning Offsets

It will be. In my experience most topics aren't compacted, so it will still
be valuable. If not difficult, leaving this empty for compacted topics to
avoid confusion will also be nice.

On Tue, Jun 26, 2018 at 4:29 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi Gwen,
>
> Thanks for the feedback.
> Regarding the partition size, couldn't "end offset - start offset" be
> misleading for compacted topics?
>
> --Vahid
>
>
>
>
> From:   Gwen Shapira 
> To: dev 
> Date:   06/26/2018 02:36 PM
> Subject:Re: [DISCUSS] KIP-325: Extend Consumer Group Command to
> Show Beginning Offsets
>
>
>
> Small suggestion: you can also add a "partition size" column - difference
> between log-end and log-start. We've had users ask for this.
>
> On Tue, Jun 26, 2018 at 2:34 PM, Gwen Shapira  wrote:
>
> > This will be useful! Thank you :)
> >
> > On Tue, Jun 26, 2018 at 11:23 AM, Vahid S Hashemian <
> > vahidhashem...@us.ibm.com> wrote:
> >
> >> Hi everyone,
> >>
> >> I have created a trivial KIP to improve the offset reporting of the
> >> consumer group command:
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-325%
>
> >> 3A+Extend+Consumer+Group+Command+to+Show+Beginning+Offsets
> >> Looking forward to your feedback!
> >>
> >> Thanks.
> >> --Vahid
> >>
> >>
> >>
> >
> >
> > --
> > *Gwen Shapira*
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter <
> https://twitter.com/ConfluentInc
> > | blog
> > <
> http://www.confluent.io/blog
> >
> >
> >
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter <
> https://twitter.com/ConfluentInc
> > | blog
> <
> http://www.confluent.io/blog
> >
>
>
>
>
>


-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter  | blog

Re: [DISCUSS] KIP-325: Extend Consumer Group Command to Show Beginning Offsets

2018-06-26 Thread Vahid S Hashemian

Hi Gwen,

Thanks for the feedback.
Regarding the partition size, couldn't "end offset - start offset" be 
misleading for compacted topics?

--Vahid




From:   Gwen Shapira 
To: dev 
Date:   06/26/2018 02:36 PM
Subject:Re: [DISCUSS] KIP-325: Extend Consumer Group Command to 
Show Beginning Offsets



Small suggestion: you can also add a "partition size" column - difference
between log-end and log-start. We've had users ask for this.

On Tue, Jun 26, 2018 at 2:34 PM, Gwen Shapira  wrote:

> This will be useful! Thank you :)
>
> On Tue, Jun 26, 2018 at 11:23 AM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
>> Hi everyone,
>>
>> I have created a trivial KIP to improve the offset reporting of the
>> consumer group command:
>> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-325%

>> 3A+Extend+Consumer+Group+Command+to+Show+Beginning+Offsets
>> Looking forward to your feedback!
>>
>> Thanks.
>> --Vahid
>>
>>
>>
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter <
https://twitter.com/ConfluentInc
> | blog
> <
http://www.confluent.io/blog
>
>
>


-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter <
https://twitter.com/ConfluentInc
> | blog
<
http://www.confluent.io/blog
>

Re: [DISCUSS] KIP-319: Replace segments with segmentSize in WindowBytesStoreSupplier

2018-06-26 Thread Matthias J. Sax

NP. Thanks!

On 6/26/18 2:54 PM, John Roesler wrote:
> Sorry for the misunderstanding, Matthias.
> 
> I have created https://issues.apache.org/jira/browse/KAFKA-7106 and
> https://issues.apache.org/jira/browse/KAFKA-7107 to track these issues.
> 
> Thanks,
> -John
> 
> On Mon, Jun 25, 2018 at 10:06 PM Matthias J. Sax 
> wrote:
> 
>> KAFKA-7080 is for this KIP.
>>
>> I meant to create a JIRA to add `segmentInterval` to `Materialized` and
>> a JIRA to add `Materialized` to `KStream#join(KStream)`.
>>
>> Thx.
>>
>>
>> -Matthias
>>
>> On 6/25/18 2:46 PM, John Roesler wrote:
>>> Ah, it turns out I did create a ticket: it's KAFKA-7080:
>>> https://issues.apache.org/jira/browse/KAFKA-7080
>>>
>>> -John
>>>
>>> On Mon, Jun 25, 2018 at 4:44 PM John Roesler  wrote:
>>>
 Matthias,

 That's a good idea. I'm not sure why I didn't...

 Thanks,
 -John

 On Mon, Jun 25, 2018 at 4:35 PM Matthias J. Sax 
 wrote:

> Ok.
>
> @John: can you create a JIRA to track this? I think KAFKA-4730 is
> related, but actually an own ticket (that is blocked by not having
> Materialized for stream-stream joins).
>
>
> -Matthias
>
> On 6/25/18 2:10 PM, Bill Bejeck wrote:
>> I agree that it makes sense to have segmentInterval as a parameter to
>> a
>> store, but I also agree with Guozhang's point about not moving as part
> of
>> this KIP.
>>
>> Thanks,
>> Bill
>>
>> On Mon, Jun 25, 2018 at 4:17 PM John Roesler 
>> wrote:
>>
>>> Thanks Matthias and Guozhang,
>>>
>>> About deprecating the "segments" field instead of making it private.
> Yes, I
>>> just took another look at the code, and that is correct. I'll update
> the
>>> KIP.
>>>
>>> I do agree that in the long run, it makes more sense as a parameter
>> to
> the
>>> store somehow than as a parameter to the window. I think this isn't a
> super
>>> high priority, though, because it's not exposed in the DSL (or it
> wasn't
>>> intended to be).
>>>
>>> I felt Guozhang's point is valid, and that we should probably revisit
> it
>>> later, possibly in the scope of
>>> https://issues.apache.org/jira/browse/KAFKA-4730
>>>
>>> I'll wait an hour or so for more feedback before moving on to a vote.
>>>
>>> Thanks again,
>>> -John
>>>
>>> On Mon, Jun 25, 2018 at 12:20 PM Guozhang Wang 
> wrote:
>>>
 Re `segmentInterval` parameter in Windows: currently it is used in
>> two
 places, the windowed stream aggregation, and the stream-stream
>> joins.
> For
 the former, we can potentially move the parameter from windowedBy()
>> to
 Materialized, but for the latter we currently do not expose a
>>> Materialized
 object yet, only the Windows spec. So I think in this KIP we
>> probably
 cannot move it immediately.

 But in future KIPs if we decide to expose the stream-stream join's
> store
>>> /
 changelog / repartition topic names, we may well adding the
> Materialized
 object into the operator, and we can then move the parameter to
 Materialized by then.


 Guozhang

 On Sun, Jun 24, 2018 at 5:16 PM, Matthias J. Sax <
> matth...@confluent.io>
 wrote:

> Thanks for the KIP. Overall, I think it makes sense to clean up the
>>> API.
>
> Couple of comments:
>
>> Sadly there's no way to "deprecate" this
>> exposure
>
> I disagree. We can just mark the variable as deprecated and I
> advocate
> to do this. When the deprecation period passed, we can make it
> private
> (or actually remove it; cf. my next comment).
>
>
> Parameter, `segmentInterval` is semantically not a "window"
> specification parameter but an implementation detail and thus a
>> store
> parameter. Would it be better to add it to `Materialized`?
>
>
> -Matthias
>
> On 6/22/18 5:13 PM, Guozhang Wang wrote:
>> Thanks John.
>>
>> On Fri, Jun 22, 2018 at 5:05 PM, John Roesler 
 wrote:
>>
>>> Thanks for the feedback, Bill and Guozhang,
>>>
>>> I've updated the KIP accordingly.
>>>
>>> Thanks,
>>> -John
>>>
>>> On Fri, Jun 22, 2018 at 5:51 PM Guozhang Wang <
>> wangg...@gmail.com>
> wrote:
>>>
 Thanks for the KIP. I'm +1 on the proposal. One minor comment on
>>> the
>>> wiki:
 the `In Windows, we will:` section code snippet is empty.

 On Fri, Jun 22, 2018 at 3:29 PM, Bill Bejeck >>
> wrote:

> Hi John,
>
> Thanks for the KIP, and overall it's a +1 for me.

Build failed in Jenkins: kafka-trunk-jdk8 #2771

See 


Changes:

[mjsax] MINOR: Fix comment in quick union (#5244)

--
[...truncated 434.42 KB...]
kafka.utils.SchedulerTest > testRestart STARTED

kafka.utils.SchedulerTest > testRestart PASSED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler STARTED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest > testPeriodicTask STARTED

kafka.utils.SchedulerTest > testPeriodicTask PASSED

kafka.utils.LoggingTest > testLog4jControllerIsRegistered STARTED

kafka.utils.LoggingTest > testLog4jControllerIsRegistered PASSED

kafka.utils.LoggingTest > testLogName STARTED

kafka.utils.LoggingTest > testLogName PASSED

kafka.utils.LoggingTest > testLogNameOverride STARTED

kafka.utils.LoggingTest > testLogNameOverride PASSED

kafka.utils.ZkUtilsTest > testGetSequenceIdMethod STARTED

kafka.utils.ZkUtilsTest > testGetSequenceIdMethod PASSED

kafka.utils.ZkUtilsTest > testAbortedConditionalDeletePath STARTED

kafka.utils.ZkUtilsTest > testAbortedConditionalDeletePath PASSED

kafka.utils.ZkUtilsTest > testGetAllPartitionsTopicWithoutPartitions STARTED

kafka.utils.ZkUtilsTest > testGetAllPartitionsTopicWithoutPartitions PASSED

kafka.utils.ZkUtilsTest > testSuccessfulConditionalDeletePath STARTED

kafka.utils.ZkUtilsTest > testSuccessfulConditionalDeletePath PASSED

kafka.utils.ZkUtilsTest > testPersistentSequentialPath STARTED

kafka.utils.ZkUtilsTest > testPersistentSequentialPath PASSED

kafka.utils.ZkUtilsTest > testClusterIdentifierJsonParsing STARTED

kafka.utils.ZkUtilsTest > testClusterIdentifierJsonParsing PASSED

kafka.utils.ZkUtilsTest > testGetLeaderIsrAndEpochForPartition STARTED

kafka.utils.ZkUtilsTest > testGetLeaderIsrAndEpochForPartition PASSED

kafka.utils.CoreUtilsTest > testGenerateUuidAsBase64 STARTED

kafka.utils.CoreUtilsTest > testGenerateUuidAsBase64 PASSED

kafka.utils.CoreUtilsTest > testAbs STARTED

kafka.utils.CoreUtilsTest > testAbs PASSED

kafka.utils.CoreUtilsTest > testReplaceSuffix STARTED

kafka.utils.CoreUtilsTest > testReplaceSuffix PASSED

kafka.utils.CoreUtilsTest > testCircularIterator STARTED

kafka.utils.CoreUtilsTest > testCircularIterator PASSED

kafka.utils.CoreUtilsTest > testReadBytes STARTED

kafka.utils.CoreUtilsTest > testReadBytes PASSED

kafka.utils.CoreUtilsTest > testCsvList STARTED

kafka.utils.CoreUtilsTest > testCsvList PASSED

kafka.utils.CoreUtilsTest > testReadInt STARTED

kafka.utils.CoreUtilsTest > testReadInt PASSED

kafka.utils.CoreUtilsTest > testAtomicGetOrUpdate STARTED

kafka.utils.CoreUtilsTest > testAtomicGetOrUpdate PASSED

kafka.utils.CoreUtilsTest > testUrlSafeBase64EncodeUUID STARTED

kafka.utils.CoreUtilsTest > testUrlSafeBase64EncodeUUID PASSED

kafka.utils.CoreUtilsTest > testCsvMap STARTED

kafka.utils.CoreUtilsTest > testCsvMap PASSED

kafka.utils.CoreUtilsTest > testInLock STARTED

kafka.utils.CoreUtilsTest > testInLock PASSED

kafka.utils.CoreUtilsTest > testTryAll STARTED

kafka.utils.CoreUtilsTest > testTryAll PASSED

kafka.utils.CoreUtilsTest > testSwallow STARTED

kafka.utils.CoreUtilsTest > testSwallow PASSED

kafka.utils.IteratorTemplateTest > testIterator STARTED

kafka.utils.IteratorTemplateTest > testIterator PASSED

kafka.utils.json.JsonValueTest > testJsonObjectIterator STARTED

kafka.utils.json.JsonValueTest > testJsonObjectIterator PASSED

kafka.utils.json.JsonValueTest > testDecodeLong STARTED

kafka.utils.json.JsonValueTest > testDecodeLong PASSED

kafka.utils.json.JsonValueTest > testAsJsonObject STARTED

kafka.utils.json.JsonValueTest > testAsJsonObject PASSED

kafka.utils.json.JsonValueTest > testDecodeDouble STARTED

kafka.utils.json.JsonValueTest > testDecodeDouble PASSED

kafka.utils.json.JsonValueTest > testDecodeOption STARTED

kafka.utils.json.JsonValueTest > testDecodeOption PASSED

kafka.utils.json.JsonValueTest > testDecodeString STARTED

kafka.utils.json.JsonValueTest > testDecodeString PASSED

kafka.utils.json.JsonValueTest > testJsonValueToString STARTED

kafka.utils.json.JsonValueTest > testJsonValueToString PASSED

kafka.utils.json.JsonValueTest > testAsJsonObjectOption STARTED

kafka.utils.json.JsonValueTest > testAsJsonObjectOption PASSED

kafka.utils.json.JsonValueTest > testAsJsonArrayOption STARTED

kafka.utils.json.JsonValueTest > testAsJsonArrayOption PASSED

kafka.utils.json.JsonValueTest > testAsJsonArray STARTED

kafka.utils.json.JsonValueTest > testAsJsonArray PASSED

kafka.utils.json.JsonValueTest > testJsonValueHashCode STARTED

kafka.utils.json.JsonValueTest > testJsonValueHashCode PASSED

kafka.utils.json.JsonValueTest > testDecodeInt STARTED

kafka.utils.json.JsonValueTest > testDecodeInt PASSED

kafka.utils.json.JsonValueTest > testDecodeMap STARTED

kafka.utils.json.JsonValueTest > testDecodeMap PASSED

kafka.utils.json.JsonValueTest > testDecodeSeq STARTED

kafka.utils.json.JsonValueTest > testDecodeSeq PASSED

Re: [Discuss] KIP-321: Add method to get TopicNameExtractor in TopologyDescription

2018-06-26 Thread John Roesler

Sorry for the late comment,

Looking at the other pieces of TopologyDescription, I noticed that pretty
much all of the "payload" of these description nodes are strings. Should we
consider returning a string from `topicNameExtractor()` instead?

In fact, if we did that, we could consider calling `toString()` on the
extractor instead of returning the class name. This would allow authors of
the extractors to provide more information about the extractor than just
its name. This might be especially useful in the case of anonymous
implementations.

Thanks for the KIP,
-John

On Mon, Jun 25, 2018 at 11:52 PM Ted Yu  wrote:

> My previous response was talking about the new method in
> InternalTopologyBuilder.
> The exception just means there is no uniform extractor for all the sinks.
>
> On Mon, Jun 25, 2018 at 8:02 PM, Matthias J. Sax 
> wrote:
>
> > Ted,
> >
> > Why? Each sink can have a different TopicNameExtractor.
> >
> >
> > -Matthias
> >
> > On 6/25/18 5:19 PM, Ted Yu wrote:
> > > If there are different TopicNameExtractor classes from multiple sink
> > nodes,
> > > the new method should throw exception alerting user of such scenario.
> > >
> > >
> > > On Mon, Jun 25, 2018 at 2:23 PM, Bill Bejeck 
> wrote:
> > >
> > >> Thanks for the KIP!
> > >>
> > >> Overall I'm +1 on the KIP.   I have one question.
> > >>
> > >> The KIP states that the method "topicNameExtractor()" is added to the
> > >> InternalTopologyBuilder.java.
> > >>
> > >> It could be that I'm missing something, but wow does this work if a
> user
> > >> has provided different TopicNameExtractor instances to multiple sink
> > nodes?
> > >>
> > >> Thanks,
> > >> Bill
> > >>
> > >>
> > >>
> > >> On Mon, Jun 25, 2018 at 1:25 PM Guozhang Wang 
> > wrote:
> > >>
> > >>> Yup I agree, generally speaking the `toString()` output is not
> > >> recommended
> > >>> to be relied on programmatically in user's code, but we've observed
> > >>> convenience-beats-any-other-reasons again and again in development
> > >>> unfortunately. I think we should still not claiming it is part of the
> > >>> public APIs that would not be changed anyhow in the future, but just
> > >>> mentioning it in the wiki for people to be aware is fine.
> > >>>
> > >>>
> > >>> Guozhang
> > >>>
> > >>> On Sun, Jun 24, 2018 at 5:01 PM, Matthias J. Sax <
> > matth...@confluent.io>
> > >>> wrote:
> > >>>
> >  Thanks for the KIP!
> > 
> >  I am don't have any further comments.
> > 
> >  For Guozhang's comment: if we mention anything about `toString()`,
> we
> >  should make explicit that `toString()` output is still not public
> >  contract and users should not rely on the output.
> > 
> >  Furhtermore, for the actual uses output, I would replace "topic:" by
> >  "extractor class:" to make the difference obvious.
> > 
> >  I am just hoping that people actually to not rely on `toString()`
> what
> >  defeats the purpose to the `TopologyDescription` class that was
> >  introduced to avoid the dependency... (Just a side comment, not
> really
> >  related to this KIP proposal itself).
> > 
> > 
> >  If there are no further comments in the next days, feel free to
> start
> >  the VOTE and open a PR.
> > 
> > 
> > 
> > 
> >  -Matthias
> > 
> >  On 6/22/18 6:04 PM, Guozhang Wang wrote:
> > > Thanks for writing the KIP!
> > >
> > > I'm +1 on the proposed changes over all. One minor suggestion: we
> > >>> should
> > > also mention that the `Sink#toString` will also be updated, in a
> way
> > >>> that
> > > if `topic()` returns null, use the other call, etc. This is because
> > > although we do not explicitly state the following logic as public
> >  protocols:
> > >
> > > ```
> > >
> > > "Sink: " + name + " (topic: " + topic() + ")\n  <-- " +
> > > nodeNames(predecessors);
> > >
> > >
> > > ```
> > >
> > > There are already some users that rely on
> > >>> `topology.describe().toString(
> >  )`
> > > to have runtime checks on the returned string values, so changing
> > >> this
> > > means that their app will break and hence need to make code
> changes.
> > >
> > > Guozhang
> > >
> > > On Wed, Jun 20, 2018 at 7:20 PM, Nishanth Pradeep <
> > >>> nishanth...@gmail.com
> > >
> > > wrote:
> > >
> > >> Hello Everyone,
> > >>
> > >> I have created a new KIP to discuss extending TopologyDescription.
> > >> You
> >  can
> > >> find it here:
> > >>
> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >> 321%3A+Add+method+to+get+TopicNameExtractor+in+TopologyDescription
> > >>
> > >> Please provide any feedback that you might have.
> > >>
> > >> Best,
> > >> Nishanth Pradeep
> > >>
> > >
> > >
> > >
> > 
> > 
> > >>>
> > >>>
> > >>> --
> > >>> -- Guozhang
> > >>>
> > >>
> > >
> >
> >
>

Re: [DISCUSS] KIP-263: Allow broker to skip sanity check of inactive segments on broker startup

2018-06-26 Thread Jason Gustafson

Hey Dong,

Sorry for being slow to catch up to this.

I think the benefit of the sanity check seems a little dubious in the first
place. We detect garbage at the end of the index file, but that's about it.
Is there any reason to think that corruption is more likely to occur there
or any other reason to think this check is still beneficial for flushed
data? I assume we did the check because we presumed it was cheap, but
perhaps the cost is adding up as the number of partitions grows. How much
does startup time improve if we skip the sanity check for data earlier than
the recovery point? Does the lazy loading itself give some additional
benefit beyond skipping the sanity check? As Jay mentions above, the sanity
checks seem strictly speaking optional. We don't bother checking the
segments themselves for example.

Thanks,
Jason




It probably still makes sense for segments beyond the recovery point

On Wed, Mar 21, 2018 at 9:59 PM, Dong Lin  wrote:

> Hey Jay,
>
> Yeah our existing sanity check only read the last entry in the index files.
> I must have miscommunicated if I previously said it was reading the full
> index. Broker appears to be spending a lot of time just to read the last
> entry of index files for every log segment. This is probably because OS
> will load a chunk of data that is much larger than the entry itself from
> disk to page cache. This KIP tries to make this part of operation lazy. I
> guess you are suggesting that we should just make the lazy loading the
> default behavior?
>
> Yes we currently require manual intervention if the log file is corrupted,
> i.e. if two messages with the same offset are appended to the disk
> (KAFKA-6488). The sanity check on broker startup is a bit different since
> it deals with the corruption of index files (e.g. offset index, time index
> and snapshot files) instead of the log data. In this case if index files
> are corrupted broker will automatically recover it by rebuilding the index
> files using data in the log files, without requiring manual intervention.
> Thus the design question is whether this should be done before broker can
> become leader for any partitions -- there is tradeoff between broker
> startup time and risk of delaying user requests if broker need to rebuild
> index files when it is already leader. I prefer lazy loading to reduce
> broker startup time. Not sure what are the feedback from the community on
> this issue.
>
> Thanks,
> Dong
>
>
> On Wed, Mar 21, 2018 at 7:36 AM, Jay Kreps  wrote:
>
> > Hey Dong,
> >
> > Makes total sense. What I'm saying is I don't think that the sanity check
> > is part of any formal guarantee we provide. It is true that corruption of
> > data flushed to disk will be a potential problem, but I don't think the
> > sanity check solves that it just has a couple heuristics to help detect
> > certain possible instances of it, right? In general I think our
> assumption
> > has been that flushed data doesn't disappear or get corrupted and if it
> > does you need to manually intervene. I don't think people want to
> configure
> > things at this level so what I was suggesting was understanding why the
> > sanity check is slow and trying to avoid that rather than making it
> > configurable. I think you mentioned it was reading the full index into
> > memory. Based on the performance you describe this could be true, but it
> > definitely should not be reading anything but the last entry in the index
> > so that would be a bug. That read also happens in sanityCheck() only in
> the
> > time-based index right? In the offset index we do the same read but it
> > happens in initialization. If that read is the slow thing it might make
> > sense to try to remove it or make it lazy in both cases. If it is some
> > other part of the code then (e.g. the size check) then that may be able
> to
> > be avoided entirely (I think by the time we sanity check we already know
> > the file size from the mapping...). That was what I meant by doing some
> > data driven analysis. Maybe a quick run with hprof would help determine
> the
> > root cause of why sanityCheck is slow?
> >
> > -Jay
> >
> > On Tue, Mar 20, 2018 at 12:13 AM Dong Lin  wrote:
> >
> > > Hey Jay,
> > >
> > > Thanks for your comments!
> > >
> > > Yeah recovery is different from the sanity check. They are correlated
> in
> > > the sense that there may still be corrupted index files even after
> clean
> > > broker shutdown. And in that case if we delay the sanity check then we
> > may
> > > delay the log recovery. The main goal of this KIP is to optimize the
> > sanity
> > > check related work so that it does not delay the broker startup much.
> > >
> > > The KIP mentioned that the sanity check is done using log recovery
> > > background thread. The name "recovery" is mentioned mainly because the
> > > background thread number is determined using the existing
> > > config num.recovery.threads.per.data.dir. I have updated the KIP to
> make
> > > this less confusing.
> >

Re: [DISCUSS] KIP-319: Replace segments with segmentSize in WindowBytesStoreSupplier

2018-06-26 Thread John Roesler

Sorry for the misunderstanding, Matthias.

I have created https://issues.apache.org/jira/browse/KAFKA-7106 and
https://issues.apache.org/jira/browse/KAFKA-7107 to track these issues.

Thanks,
-John

On Mon, Jun 25, 2018 at 10:06 PM Matthias J. Sax 
wrote:

> KAFKA-7080 is for this KIP.
>
> I meant to create a JIRA to add `segmentInterval` to `Materialized` and
> a JIRA to add `Materialized` to `KStream#join(KStream)`.
>
> Thx.
>
>
> -Matthias
>
> On 6/25/18 2:46 PM, John Roesler wrote:
> > Ah, it turns out I did create a ticket: it's KAFKA-7080:
> > https://issues.apache.org/jira/browse/KAFKA-7080
> >
> > -John
> >
> > On Mon, Jun 25, 2018 at 4:44 PM John Roesler  wrote:
> >
> >> Matthias,
> >>
> >> That's a good idea. I'm not sure why I didn't...
> >>
> >> Thanks,
> >> -John
> >>
> >> On Mon, Jun 25, 2018 at 4:35 PM Matthias J. Sax 
> >> wrote:
> >>
> >>> Ok.
> >>>
> >>> @John: can you create a JIRA to track this? I think KAFKA-4730 is
> >>> related, but actually an own ticket (that is blocked by not having
> >>> Materialized for stream-stream joins).
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 6/25/18 2:10 PM, Bill Bejeck wrote:
>  I agree that it makes sense to have segmentInterval as a parameter to
> a
>  store, but I also agree with Guozhang's point about not moving as part
> >>> of
>  this KIP.
> 
>  Thanks,
>  Bill
> 
>  On Mon, Jun 25, 2018 at 4:17 PM John Roesler 
> wrote:
> 
> > Thanks Matthias and Guozhang,
> >
> > About deprecating the "segments" field instead of making it private.
> >>> Yes, I
> > just took another look at the code, and that is correct. I'll update
> >>> the
> > KIP.
> >
> > I do agree that in the long run, it makes more sense as a parameter
> to
> >>> the
> > store somehow than as a parameter to the window. I think this isn't a
> >>> super
> > high priority, though, because it's not exposed in the DSL (or it
> >>> wasn't
> > intended to be).
> >
> > I felt Guozhang's point is valid, and that we should probably revisit
> >>> it
> > later, possibly in the scope of
> > https://issues.apache.org/jira/browse/KAFKA-4730
> >
> > I'll wait an hour or so for more feedback before moving on to a vote.
> >
> > Thanks again,
> > -John
> >
> > On Mon, Jun 25, 2018 at 12:20 PM Guozhang Wang 
> >>> wrote:
> >
> >> Re `segmentInterval` parameter in Windows: currently it is used in
> two
> >> places, the windowed stream aggregation, and the stream-stream
> joins.
> >>> For
> >> the former, we can potentially move the parameter from windowedBy()
> to
> >> Materialized, but for the latter we currently do not expose a
> > Materialized
> >> object yet, only the Windows spec. So I think in this KIP we
> probably
> >> cannot move it immediately.
> >>
> >> But in future KIPs if we decide to expose the stream-stream join's
> >>> store
> > /
> >> changelog / repartition topic names, we may well adding the
> >>> Materialized
> >> object into the operator, and we can then move the parameter to
> >> Materialized by then.
> >>
> >>
> >> Guozhang
> >>
> >> On Sun, Jun 24, 2018 at 5:16 PM, Matthias J. Sax <
> >>> matth...@confluent.io>
> >> wrote:
> >>
> >>> Thanks for the KIP. Overall, I think it makes sense to clean up the
> > API.
> >>>
> >>> Couple of comments:
> >>>
>  Sadly there's no way to "deprecate" this
>  exposure
> >>>
> >>> I disagree. We can just mark the variable as deprecated and I
> >>> advocate
> >>> to do this. When the deprecation period passed, we can make it
> >>> private
> >>> (or actually remove it; cf. my next comment).
> >>>
> >>>
> >>> Parameter, `segmentInterval` is semantically not a "window"
> >>> specification parameter but an implementation detail and thus a
> store
> >>> parameter. Would it be better to add it to `Materialized`?
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 6/22/18 5:13 PM, Guozhang Wang wrote:
>  Thanks John.
> 
>  On Fri, Jun 22, 2018 at 5:05 PM, John Roesler 
> >> wrote:
> 
> > Thanks for the feedback, Bill and Guozhang,
> >
> > I've updated the KIP accordingly.
> >
> > Thanks,
> > -John
> >
> > On Fri, Jun 22, 2018 at 5:51 PM Guozhang Wang <
> wangg...@gmail.com>
> >>> wrote:
> >
> >> Thanks for the KIP. I'm +1 on the proposal. One minor comment on
> > the
> > wiki:
> >> the `In Windows, we will:` section code snippet is empty.
> >>
> >> On Fri, Jun 22, 2018 at 3:29 PM, Bill Bejeck  >
> >>> wrote:
> >>
> >>> Hi John,
> >>>
> >>> Thanks for the KIP, and overall it's a +1 for me.
> >>>
> >>> In the JavaDoc for the segmentInterval method, there's no
> mention
>

[jira] [Created] (KAFKA-7107) Ability to configure state store for JoinWindows in KStream-KStream join

2018-06-26 Thread John Roesler (JIRA)

John Roesler created KAFKA-7107:
---

 Summary: Ability to configure state store for JoinWindows in 
KStream-KStream join
 Key: KAFKA-7107
 URL: https://issues.apache.org/jira/browse/KAFKA-7107
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: John Roesler


Currently, the KStream-KStream join operation internally provisions window 
stores to support the JoinWindow configuration.

 

However, unlike all the other stateful processors, it does not allow 
configuration of the stores. We should consider adding DSL methods taking 
Materialized configs for these stores.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7106) Remove segment/segmentInterval from Window definition

2018-06-26 Thread John Roesler (JIRA)

John Roesler created KAFKA-7106:
---

 Summary: Remove segment/segmentInterval from Window definition
 Key: KAFKA-7106
 URL: https://issues.apache.org/jira/browse/KAFKA-7106
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: John Roesler


Currently, Window configures segment and segmentInterval properties, but these 
aren't truly properties of a window in general.

Rather, they are properties of the particular implementation that we currently 
have: a segmented store. Therefore, these properties should be moved to 
configure only that implementation.

 

This may be related to KAFKA-4730, since an in-memory window store wouldn't 
necessarily need to be segmented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-325: Extend Consumer Group Command to Show Beginning Offsets

Small suggestion: you can also add a "partition size" column - difference
between log-end and log-start. We've had users ask for this.

On Tue, Jun 26, 2018 at 2:34 PM, Gwen Shapira  wrote:

> This will be useful! Thank you :)
>
> On Tue, Jun 26, 2018 at 11:23 AM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
>> Hi everyone,
>>
>> I have created a trivial KIP to improve the offset reporting of the
>> consumer group command:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-325%
>> 3A+Extend+Consumer+Group+Command+to+Show+Beginning+Offsets
>> Looking forward to your feedback!
>>
>> Thanks.
>> --Vahid
>>
>>
>>
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter  | blog
> 
>
>


-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter  | blog

Re: [DISCUSS] KIP-325: Extend Consumer Group Command to Show Beginning Offsets

This will be useful! Thank you :)

On Tue, Jun 26, 2018 at 11:23 AM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi everyone,
>
> I have created a trivial KIP to improve the offset reporting of the
> consumer group command:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 325%3A+Extend+Consumer+Group+Command+to+Show+Beginning+Offsets
> Looking forward to your feedback!
>
> Thanks.
> --Vahid
>
>
>


-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter  | blog

Re: [VOTE] KIP-324: Add method to get metrics() in AdminClient

2018-06-26 Thread Guozhang Wang

+1. Thanks.

On Tue, Jun 26, 2018 at 2:31 PM, Yishun Guan  wrote:

> Hi All,
>
> I am starting a vote on this KIP:
>
> https://cwiki.apache.org/confluence/x/lQg0BQ
>
> Thanks,
> Yishun
>



-- 
-- Guozhang

[VOTE] KIP-324: Add method to get metrics() in AdminClient

Hi All,

I am starting a vote on this KIP:

https://cwiki.apache.org/confluence/x/lQg0BQ

Thanks,
Yishun

Build failed in Jenkins: kafka-trunk-jdk8 #2770

See 


Changes:

[wangguoz] MINOR: Add note about num.standby.replicas (#5271)

--
[...truncated 435.06 KB...]
kafka.utils.SchedulerTest > testRestart STARTED

kafka.utils.SchedulerTest > testRestart PASSED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler STARTED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest > testPeriodicTask STARTED

kafka.utils.SchedulerTest > testPeriodicTask PASSED

kafka.utils.LoggingTest > testLog4jControllerIsRegistered STARTED

kafka.utils.LoggingTest > testLog4jControllerIsRegistered PASSED

kafka.utils.LoggingTest > testLogName STARTED

kafka.utils.LoggingTest > testLogName PASSED

kafka.utils.LoggingTest > testLogNameOverride STARTED

kafka.utils.LoggingTest > testLogNameOverride PASSED

kafka.utils.ZkUtilsTest > testGetSequenceIdMethod STARTED

kafka.utils.ZkUtilsTest > testGetSequenceIdMethod PASSED

kafka.utils.ZkUtilsTest > testAbortedConditionalDeletePath STARTED

kafka.utils.ZkUtilsTest > testAbortedConditionalDeletePath PASSED

kafka.utils.ZkUtilsTest > testGetAllPartitionsTopicWithoutPartitions STARTED

kafka.utils.ZkUtilsTest > testGetAllPartitionsTopicWithoutPartitions PASSED

kafka.utils.ZkUtilsTest > testSuccessfulConditionalDeletePath STARTED

kafka.utils.ZkUtilsTest > testSuccessfulConditionalDeletePath PASSED

kafka.utils.ZkUtilsTest > testPersistentSequentialPath STARTED

kafka.utils.ZkUtilsTest > testPersistentSequentialPath PASSED

kafka.utils.ZkUtilsTest > testClusterIdentifierJsonParsing STARTED

kafka.utils.ZkUtilsTest > testClusterIdentifierJsonParsing PASSED

kafka.utils.ZkUtilsTest > testGetLeaderIsrAndEpochForPartition STARTED

kafka.utils.ZkUtilsTest > testGetLeaderIsrAndEpochForPartition PASSED

kafka.utils.CoreUtilsTest > testGenerateUuidAsBase64 STARTED

kafka.utils.CoreUtilsTest > testGenerateUuidAsBase64 PASSED

kafka.utils.CoreUtilsTest > testAbs STARTED

kafka.utils.CoreUtilsTest > testAbs PASSED

kafka.utils.CoreUtilsTest > testReplaceSuffix STARTED

kafka.utils.CoreUtilsTest > testReplaceSuffix PASSED

kafka.utils.CoreUtilsTest > testCircularIterator STARTED

kafka.utils.CoreUtilsTest > testCircularIterator PASSED

kafka.utils.CoreUtilsTest > testReadBytes STARTED

kafka.utils.CoreUtilsTest > testReadBytes PASSED

kafka.utils.CoreUtilsTest > testCsvList STARTED

kafka.utils.CoreUtilsTest > testCsvList PASSED

kafka.utils.CoreUtilsTest > testReadInt STARTED

kafka.utils.CoreUtilsTest > testReadInt PASSED

kafka.utils.CoreUtilsTest > testAtomicGetOrUpdate STARTED

kafka.utils.CoreUtilsTest > testAtomicGetOrUpdate PASSED

kafka.utils.CoreUtilsTest > testUrlSafeBase64EncodeUUID STARTED

kafka.utils.CoreUtilsTest > testUrlSafeBase64EncodeUUID PASSED

kafka.utils.CoreUtilsTest > testCsvMap STARTED

kafka.utils.CoreUtilsTest > testCsvMap PASSED

kafka.utils.CoreUtilsTest > testInLock STARTED

kafka.utils.CoreUtilsTest > testInLock PASSED

kafka.utils.CoreUtilsTest > testTryAll STARTED

kafka.utils.CoreUtilsTest > testTryAll PASSED

kafka.utils.CoreUtilsTest > testSwallow STARTED

kafka.utils.CoreUtilsTest > testSwallow PASSED

kafka.utils.IteratorTemplateTest > testIterator STARTED

kafka.utils.IteratorTemplateTest > testIterator PASSED

kafka.utils.json.JsonValueTest > testJsonObjectIterator STARTED

kafka.utils.json.JsonValueTest > testJsonObjectIterator PASSED

kafka.utils.json.JsonValueTest > testDecodeLong STARTED

kafka.utils.json.JsonValueTest > testDecodeLong PASSED

kafka.utils.json.JsonValueTest > testAsJsonObject STARTED

kafka.utils.json.JsonValueTest > testAsJsonObject PASSED

kafka.utils.json.JsonValueTest > testDecodeDouble STARTED

kafka.utils.json.JsonValueTest > testDecodeDouble PASSED

kafka.utils.json.JsonValueTest > testDecodeOption STARTED

kafka.utils.json.JsonValueTest > testDecodeOption PASSED

kafka.utils.json.JsonValueTest > testDecodeString STARTED

kafka.utils.json.JsonValueTest > testDecodeString PASSED

kafka.utils.json.JsonValueTest > testJsonValueToString STARTED

kafka.utils.json.JsonValueTest > testJsonValueToString PASSED

kafka.utils.json.JsonValueTest > testAsJsonObjectOption STARTED

kafka.utils.json.JsonValueTest > testAsJsonObjectOption PASSED

kafka.utils.json.JsonValueTest > testAsJsonArrayOption STARTED

kafka.utils.json.JsonValueTest > testAsJsonArrayOption PASSED

kafka.utils.json.JsonValueTest > testAsJsonArray STARTED

kafka.utils.json.JsonValueTest > testAsJsonArray PASSED

kafka.utils.json.JsonValueTest > testJsonValueHashCode STARTED

kafka.utils.json.JsonValueTest > testJsonValueHashCode PASSED

kafka.utils.json.JsonValueTest > testDecodeInt STARTED

kafka.utils.json.JsonValueTest > testDecodeInt PASSED

kafka.utils.json.JsonValueTest > testDecodeMap STARTED

kafka.utils.json.JsonValueTest > testDecodeMap PASSED

kafka.utils.json.JsonValueTest > testDecodeSeq STARTED

kafka.utils.json.JsonValueTest > testDecodeSeq

[jira] [Created] (KAFKA-7105) Refactor RocksDBSegmentsBatchingRestoreCallback and RocksDBBatchingRestoreCallback into a single class

2018-06-26 Thread Liquan Pei (JIRA)

Liquan Pei created KAFKA-7105:
-

 Summary: Refactor RocksDBSegmentsBatchingRestoreCallback and 
RocksDBBatchingRestoreCallback into a single class
 Key: KAFKA-7105
 URL: https://issues.apache.org/jira/browse/KAFKA-7105
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Affects Versions: 1.1.0
Reporter: Liquan Pei
Assignee: Liquan Pei






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7104) ReplicaFetcher thread may die because of inconsistent log start offset in fetch response

2018-06-26 Thread Anna Povzner (JIRA)

Anna Povzner created KAFKA-7104:
---

 Summary: ReplicaFetcher thread may die because of inconsistent log 
start offset in fetch response
 Key: KAFKA-7104
 URL: https://issues.apache.org/jira/browse/KAFKA-7104
 Project: Kafka
  Issue Type: Bug
Affects Versions: 1.1.0, 1.0.0
Reporter: Anna Povzner
Assignee: Anna Povzner


What we saw:

The follower fetches offset 116617, which it was able successfully append. 
However, leader's log start offset in fetch request was 116753, which was 
higher than fetched offset 116617. When replica fetcher thread tried to 
increment log start offset to leader's log start offset, it failed with 
OffsetOutOfRangeException: 

[2018-06-23 00:45:37,409] ERROR [ReplicaFetcher replicaId=1002, leaderId=1001, 
fetcherId=0] Error due to (kafka.server.ReplicaFetcherThread) 
 kafka.common.KafkaException: Error processing data for partition X-N offset 
116617 

Caused by: org.apache.kafka.common.errors.OffsetOutOfRangeException: Cannot 
increment the log start offset to 116753 of partition X-N since it is larger 
than the high watermark 116619

 

In leader's log, we see that log start offset was incremented almost at the 
same time (within one 100 ms or so). 

[2018-06-23 00:45:37,339] INFO Incrementing log start offset of partition X-N 
to 116753 in dir /kafka/kafka-logs (kafka.log.Log)

 

In leader's logic: ReplicaManager#ReplicaManager first calls readFromLocalLog() 
that reads from local log and returns LogReadResult that contains fetched data 
and leader's log start offset and HW. However, it then calls 
ReplicaManager#updateFollowerLogReadResults() that may move leader's log start 
offset and update leader's log start offset and HW in fetch response. If 
deleteRecords() happens in between, it is possible that log start offset may 
move beyond fetched offset. As a result, fetch response will contain fetched 
data but log start offset that is beyond fetched offset (and indicate the state 
on leader that fetched data does not actually exist anymore on leader).

When a follower receives such fetch response, it will first append, then move 
it's HW no further than its LEO, which maybe less than leader's log start 
offset in fetch response, and then call 
`replica.maybeIncrementLogStartOffset(leaderLogStartOffset)` which will throw 
OffsetOutOfRangeException exception causing the fetcher thread to stop. 

 

*Suggested fix:*

If the leader moves log start offset beyond fetched offset, 
ReplicaManager#updateFollowerLogReadResults()  should update the log read 
result with OFFSET_OUT_OF_RANGE error, which will cause the follower to reset 
fetch offset to leader's log start offset.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-322: Return new error code for DeleteTopics API when topic deletion disabled.

Looks good overall.

nit: you are going to fill in an actual value below in your PR, right ?

private static final long serialVersionUID = 1L;

In Motivation, please mention the IllegalStateException scenario.

On Tue, Jun 26, 2018 at 9:34 AM, Manikumar 
wrote:

> Hi all,
>
> I have created a minor KIP to return new error code for DeleteTopics API
> when topic deletion disabled.
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87295558
>
> Please take a look.
>
> Thanks,
>

[jira] [Created] (KAFKA-7103) Use bulkloading for RocksDBSegmentedBytesStore during init

2018-06-26 Thread Liquan Pei (JIRA)

Liquan Pei created KAFKA-7103:
-

 Summary: Use bulkloading for RocksDBSegmentedBytesStore during init
 Key: KAFKA-7103
 URL: https://issues.apache.org/jira/browse/KAFKA-7103
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.1.0
Reporter: Liquan Pei
Assignee: Liquan Pei


We should use bulk loading for recovering RocksDBWindowStore, same as 
RocksDBStore. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-324: Add method to get metrics() in AdminClient

Hi Colin,

I agree with what Guozhang's opinion that because all the other clients
have it (producer, consumer..) and this will gain more visibility for those
application that use admin client. (Now I added this sentence to the KIP)

Since this returns an unmodifiableMap(like all the other client's metrics()
return), I assume this will be thread-safe, what do you think?

Thanks,
Yishun

On Tue, Jun 26, 2018 at 11:51 AM, Colin McCabe  wrote:

> Can you add a little more explanation to the KIP for why you are adding
> this method?  Is it something streams needs, for example?  Will it help
> other applications that use admin client and want to expose metrics?
>
> What are the thread-safety guarantees for the map which is returned?
>
> best,
> Colin
>
>
> On Tue, Jun 26, 2018, at 11:29, Yishun Guan wrote:
> > Hi All,
> >
> > I am starting a vote on this KIP:
> >
> > https://cwiki.apache.org/confluence/x/lQg0BQ
> >
> > Thanks,
> > Yishun
>

Re: [DISCUSS] KIP-326: Schedulable KTable as Graph source

What's the relationship between this KIP and KIP-323 ?

Thanks

On Tue, Jun 26, 2018 at 11:22 AM, Flávio Stutz 
wrote:

> Hey, guys, I've just created a new KIP about creating a new DSL graph
> source for realtime partitioned consolidations.
>
> We have faced the following scenario/problem in a lot of situations with
> KStreams:
>- Huge incoming data being processed by numerous application instances
>- Need to aggregate different fields whose records span all topic
> partitions (something like “total amount spent by people aged > 30 yrs”
> when processing a topic partitioned by userid).
>
> The challenge here is to manage this kind of situation without any
> bottlenecks. We don't need the “global aggregation” to be processed at each
> incoming message. On a scenario of 500 instances, each handling 1k
> messages/s, any single point of aggregation (single partitioned topics,
> global tables or external databases) would create a bottleneck of 500k
> messages/s for single threaded/CPU elements.
>
> For this scenario, it is possible to store the partial aggregations on
> local stores and, from time to time, query those states and aggregate them
> as a single value, avoiding bottlenecks. This is a way to create a "timed
> aggregation barrier”.
>
> If we leverage this kind of built-in feature we could greatly enhance the
> ability of KStreams to better handle the CAP Theorem characteristics, so
> that one could choose to have Consistency over Availability when needed.
>
> We started this discussion with Matthias J. Sax here:
> https://issues.apache.org/jira/browse/KAFKA-6953
>
> If you want to see more, go to KIP-326 at:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 326%3A+Schedulable+KTable+as+Graph+source
>
> -Flávio Stutz
>

[DISCUSS] KIP-328: Ability to suppress updates for KTables

2018-06-26 Thread John Roesler

Hello devs and users,

Please take some time to consider this proposal for Kafka Streams:

KIP-328: Ability to suppress updates for KTables

link: https://cwiki.apache.org/confluence/x/sQU0BQ

The basic idea is to provide:
* more usable control over update rate (vs the current state store caches)
* the final-result-for-windowed-computations feature which several people
have requested

I look forward to your feedback!

Thanks,
-John

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

2018-06-26 Thread Jason Gustafson

The other thing I forgot to mention is that resetting the offset using the
leader epoch is only available with the latest message format. By
supporting reset by timestamp, users on the v1 format can still get some
benefit from this KIP.

-Jason

On Tue, Jun 26, 2018 at 11:47 AM, Jason Gustafson 
wrote:

> Hey Dong,
>
> Thanks for the comments.
>
> - The KIP says that, with auto.offset.reset="closest", timestamp is used to
>> find offset if truncation offset is unknown. It seems that if consumer
>> knows the timestamp of the last message, then the consumer should also
>> know
>> the (offset, leaderEpoch) of the last message which can then be used for
>> find the truncation offset. Can you explain why truncation offset is
>> unknown in this case?
>
>
> The intent of the new reset policy is to automatically locate the closest
> offset within the limits of Kafka log semantics. Unlike replicas,
> consumers do not know the full history of leader epochs that have been
> previously read. In some scenarios, they may not be able to precisely find
> the offset where the log diverged after a sequence of unclean leader
> elections (see KIP-279 for more detail). It seemed unfortunate in these
> cases to have to resort to the coarse-grained resetting using either the
> earliest or latest offset. Using the timestamp, we can find a more accurate
> reset point and minimize the amount of loss or duplication.
>
> - How does consumer differentiates between "Offset out of rnage (too low)"
>> and "Offset out of range (unknown truncation offset)", i.e. the two
>> columns
>> in table provided in the KIP?
>
>
> We know when an offset is too low because we have the start offset of the
> log from the fetch response. Following this KIP, that should really be the
> only time we get an OutOfRange error (other than buggy application code).
> The other two cases are distinguished based on whether we are able to find
> the right offset of divergence.
>
> - It is probably a typo. Maybe fix "This is not the last The" in the
>> Proposed Section.
>
>
> Thanks. Magnus noticed this too and I fixed it earlier this morning. Good
> to know who's actually reading the proposal!
>
> -Jason
>
>
>
> On Tue, Jun 26, 2018 at 11:09 AM, Dong Lin  wrote:
>
>> Hey Jason,
>>
>> Thanks for the KIP! It is pretty useful.
>>
>> At high level the new set of reset policies may be a bit complicated and
>> confusing to users. I am wondering whether we can simplify it. A few
>> questions below:
>>
>> - The KIP says that, with auto.offset.reset="closest", timestamp is used
>> to
>> find offset if truncation offset is unknown. It seems that if consumer
>> knows the timestamp of the last message, then the consumer should also
>> know
>> the (offset, leaderEpoch) of the last message which can then be used for
>> find the truncation offset. Can you explain why truncation offset is
>> unknown in this case?
>>
>> - How does consumer differentiates between "Offset out of rnage (too low)"
>> and "Offset out of range (unknown truncation offset)", i.e. the two
>> columns
>> in table provided in the KIP?
>>
>> - It is probably a typo. Maybe fix "This is not the last The" in the
>> Proposed Section.
>>
>>
>> Thanks,
>> Dong
>>
>> On Mon, Jun 25, 2018 at 9:17 AM, Jason Gustafson 
>> wrote:
>>
>> > Hey All,
>> >
>> > I wrote up a KIP to handle one more edge case in the replication
>> protocol
>> > and to support better handling of truncation in the consumer when
>> unclean
>> > leader election is enabled. Let me know what you think.
>> >
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-320%3A
>> > +Allow+fetchers+to+detect+and+handle+log+truncation
>> >
>> > Thanks to Anna Povzner and Dong Lin for initial feedback.
>> >
>> > Thanks,
>> > Jason
>> >
>>
>
>

Build failed in Jenkins: kafka-2.0-jdk8 #57

See 


Changes:

[lindong28] KAFKA-6949; alterReplicaLogDirs() should grab partition lock when

--
[...truncated 434.51 KB...]
kafka.zookeeper.ZooKeeperClientTest > 
testBlockOnRequestCompletionFromStateChangeHandler PASSED

kafka.zookeeper.ZooKeeperClientTest > testUnresolvableConnectString STARTED

kafka.zookeeper.ZooKeeperClientTest > testUnresolvableConnectString PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testPipelinedGetData STARTED

kafka.zookeeper.ZooKeeperClientTest > testPipelinedGetData PASSED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChildChangeHandlerForChildChange 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChildChangeHandlerForChildChange 
PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenExistingZNodeWithChildren 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenExistingZNodeWithChildren 
PASSED

kafka.zookeeper.ZooKeeperClientTest > testSetDataExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testSetDataExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testMixedPipeline STARTED

kafka.zookeeper.ZooKeeperClientTest > testMixedPipeline PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetDataExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetDataExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testDeleteExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testDeleteExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testSessionExpiry STARTED

kafka.zookeeper.ZooKeeperClientTest > testSessionExpiry PASSED

kafka.zookeeper.ZooKeeperClientTest > testSetDataNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testSetDataNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testDeleteNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testDeleteNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testExistsExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testExistsExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testZooKeeperStateChangeRateMetrics 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testZooKeeperStateChangeRateMetrics PASSED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForDeletion STARTED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForDeletion PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetAclNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetAclNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testStateChangeHandlerForAuthFailure 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testStateChangeHandlerForAuthFailure 
PASSED

kafka.network.SocketServerTest > testGracefulClose STARTED

kafka.network.SocketServerTest > testGracefulClose PASSED

kafka.network.SocketServerTest > 
testSendActionResponseWithThrottledChannelWhereThrottlingAlreadyDone STARTED

kafka.network.SocketServerTest > 
testSendActionResponseWithThrottledChannelWhereThrottlingAlreadyDone PASSED

kafka.network.SocketServerTest > controlThrowable STARTED

kafka.network.SocketServerTest > controlThrowable PASSED

kafka.network.SocketServerTest > testRequestMetricsAfterStop STARTED

kafka.network.SocketServerTest > testRequestMetricsAfterStop PASSED

kafka.network.SocketServerTest > testConnectionIdReuse STARTED

kafka.network.SocketServerTest > testConnectionIdReuse PASSED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
STARTED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
PASSED

kafka.network.SocketServerTest > testProcessorMetricsTags STARTED

kafka.network.SocketServerTest > testProcessorMetricsTags PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIp STARTED

kafka.network.SocketServerTest > testMaxConnectionsPerIp PASSED

kafka.network.SocketServerTest > testConnectionId STARTED

kafka.network.SocketServerTest > testConnectionId PASSED

kafka.network.SocketServerTest > 
testBrokerSendAfterChannelClosedUpdatesRequestMetrics STARTED

kafka.network.SocketServerTest > 
testBrokerSendAfterChannelClosedUpdatesRequestMetrics PASSED

kafka.network.SocketServerTest > testNoOpAction STARTED

kafka.network.SocketServerTest > testNoOpAction PASSED

kafka.network.SocketServerTest > simpleRequest STARTED

kafka.network.SocketServerTest > simpleRequest PASSED

kafka.network.SocketServerTest > closingChannelException STARTED

kafka.network.SocketServerTest > closingChannelException PASSED

kafka.network.SocketServerTest > 
testSendActionResponseWithThrottledChannelWhereThrottlingInProgress STARTED

kafka.network.SocketServerTest > 
testSendActionResponseWithThrottledChannelWhereThrottlingInProgress PASSED

kafka.network.SocketServerTest >

Re: SASL Unit test failing

2018-06-26 Thread Rajini Sivaram

Colin/Ahmed,

Can you run with debug logging turned on and attach the logs? You could
just change log level in clients/src/test/resources/log4j.properties and
run the tests.

Thanks,

Rajini


On Tue, Jun 26, 2018 at 7:40 PM, Colin McCabe  wrote:

> On trunk, testMultipleServerMechanisms failed for me, as well as
> testAuthenticateCallbackHandlerMechanisms and testMechanismPluggability.
>
> org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest >
> testMultipleServerMechanisms FAILED
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at org.apache.kafka.common.network.NetworkTestUtils.
> waitForChannelReady(NetworkTestUtils.java:79)
> at org.apache.kafka.common.network.NetworkTestUtils.
> checkClientConnection(NetworkTestUtils.java:52)
> at org.apache.kafka.common.security.authenticator.
> SaslAuthenticatorTest.testMultipleServerMechanisms(
> SaslAuthenticatorTest.java:281)
>
> org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest >
> testAuthenticateCallbackHandlerMechanisms FAILED
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at org.apache.kafka.common.network.NetworkTestUtils.
> waitForChannelReady(NetworkTestUtils.java:79)
> at org.apache.kafka.common.network.NetworkTestUtils.
> checkClientConnection(NetworkTestUtils.java:52)
> at org.apache.kafka.common.security.authenticator.
> SaslAuthenticatorTest.createAndCheckClientConnection
> (SaslAuthenticatorTest.java:1475)
> at org.apache.kafka.common.security.authenticator.
> SaslAuthenticatorTest.testAuthenticateCallbackHandlerMechanisms(
> SaslAuthenticatorTest.java:776)
>
> org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest >
> testMechanismPluggability FAILED
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at org.apache.kafka.common.network.NetworkTestUtils.
> waitForChannelReady(NetworkTestUtils.java:79)
> at org.apache.kafka.common.network.NetworkTestUtils.
> checkClientConnection(NetworkTestUtils.java:52)
> at org.apache.kafka.common.security.authenticator.
> SaslAuthenticatorTest.createAndCheckClientConnection
> (SaslAuthenticatorTest.java:1475)
> at org.apache.kafka.common.security.authenticator.
> SaslAuthenticatorTest.testMechanismPluggability(
> SaslAuthenticatorTest.java:257)
>
> best,
> Colin
>
> On Mon, Jun 25, 2018, at 18:02, Ted Yu wrote:
> > I ran the test on Linux as well.
> >
> > cat /etc/redhat-release
> > CentOS Linux release 7.2.1511 (Core)
> >
> > Java version: 1.8.0_161, vendor: Oracle Corporation
> > Java home: /jdk1.8.0_161/jre
> > Default locale: en_US, platform encoding: UTF-8
> > OS name: "linux", version: "3.10.0-327.28.3.el7.x86_64", arch: "amd64",
> > family: "unix"
> >
> > On Mon, Jun 25, 2018 at 5:42 PM, Ted Yu  wrote:
> >
> > > Here was the command I used:
> > >
> > > ./gradlew -Dtest.single=SaslAuthenticatorTest clients:test
> > >
> > > On Mon, Jun 25, 2018 at 5:39 PM, Ahmed A  wrote:
> > >
> > >> I ran test with -i option as follows - "./gradlew  -i test".  The
> same set
> > >> of three tests failed.
> > >>
> > >> My environment:
> > >> $ java -version
> > >> java version "1.8.0_121"
> > >> Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
> > >> Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
> > >>
> > >> $ cat /etc/redhat-release
> > >> Red Hat Enterprise Linux Workstation release 7.3 (Maipo)
> > >> $ uname -a
> > >> Linux  ahmed  3.10.0-514.36.5.el7.x86_64 #1 SMP Thu Dec 28 21:42:18
> EST
> > >> 2017 x86_64 x86_64 x86_64 GNU/Linux
> > >>
> > >>
> > >> Can you please let me know how I can run an individual unit test, what
> > >> options do I provide?
> > >>
> > >>
> > >> Thank you,
> > >> Ahmed.
> > >>
> > >>
> > >>
> > >> On Mon, Jun 25, 2018 at 2:47 PM, Ted Yu  wrote:
> > >>
> > >> > I ran the test alone which passed.
> > >> >
> > >> > Can you include -i on the command line to see if there is some clue
> from
> > >> > the output ?
> > >> >
> > >> > Here is my environment:
> > >> >
> > >> > Java version: 1.8.0_151, vendor: Oracle Corporation
> > >> > Java home:
> > >> > /Library/Java/JavaVirtualMachines/jdk1.8.0_
> 151.jdk/Contents/Home/jre
> > >> > Default locale: en_US, platform encoding: UTF-8
> > >> > OS name: "mac os x", version: "10.11.3", arch: "x86_64", family:
> "mac"
> > >> >
> > >> > FYI
> > >> >
> > >> > On Mon, Jun 25, 2018 at 12:59 PM, Ahmed A 
> wrote:
> > >> >
> > >> > > Hello,
> > >> > >
> > >> > > I did a fresh clone of the kafka src code, and the following SASL
> unit
> > >> > > tests have been failing

[jira] [Created] (KAFKA-7102) Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@10c8df20

2018-06-26 Thread sankar (JIRA)

sankar created KAFKA-7102:
-

 Summary:  Error in fetch 
kafka.server.ReplicaFetcherThread$FetchRequest@10c8df20
 Key: KAFKA-7102
 URL: https://issues.apache.org/jira/browse/KAFKA-7102
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 0.10.1.0
Reporter: sankar
 Attachments: kafka_java_io_exception.txt

we faced  Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@10c8df20

 Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@10c8df20

We have four node kafka cluster in production environment. We experienced 
suddenly kafka connect issue across cluster. 

manual restart kafka service on all the nodes fixed the issue.

I attached the complete log. Please check the log.

kindly let me know what information more needed from my side.

Thanks in advance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-324: Add method to get metrics() in AdminClient

2018-06-26 Thread Colin McCabe

Can you add a little more explanation to the KIP for why you are adding this 
method?  Is it something streams needs, for example?  Will it help other 
applications that use admin client and want to expose metrics?

What are the thread-safety guarantees for the map which is returned?

best,
Colin

On Tue, Jun 26, 2018, at 11:29, Yishun Guan wrote:
> Hi All,
> 
> I am starting a vote on this KIP:
> 
> https://cwiki.apache.org/confluence/x/lQg0BQ
> 
> Thanks,
> Yishun

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

2018-06-26 Thread Jason Gustafson

Hey Dong,

Thanks for the comments.

- The KIP says that, with auto.offset.reset="closest", timestamp is used to
> find offset if truncation offset is unknown. It seems that if consumer
> knows the timestamp of the last message, then the consumer should also know
> the (offset, leaderEpoch) of the last message which can then be used for
> find the truncation offset. Can you explain why truncation offset is
> unknown in this case?

The intent of the new reset policy is to automatically locate the closest
offset within the limits of Kafka log semantics. Unlike replicas, consumers
do not know the full history of leader epochs that have been previously
read. In some scenarios, they may not be able to precisely find the offset
where the log diverged after a sequence of unclean leader elections (see
KIP-279 for more detail). It seemed unfortunate in these cases to have to
resort to the coarse-grained resetting using either the earliest or latest
offset. Using the timestamp, we can find a more accurate reset point and
minimize the amount of loss or duplication.

- How does consumer differentiates between "Offset out of rnage (too low)"
> and "Offset out of range (unknown truncation offset)", i.e. the two columns
> in table provided in the KIP?

We know when an offset is too low because we have the start offset of the
log from the fetch response. Following this KIP, that should really be the
only time we get an OutOfRange error (other than buggy application code).
The other two cases are distinguished based on whether we are able to find
the right offset of divergence.

- It is probably a typo. Maybe fix "This is not the last The" in the
> Proposed Section.

Thanks. Magnus noticed this too and I fixed it earlier this morning. Good
to know who's actually reading the proposal!

-Jason

On Tue, Jun 26, 2018 at 11:09 AM, Dong Lin  wrote:

> Hey Jason,
>
> Thanks for the KIP! It is pretty useful.
>
> At high level the new set of reset policies may be a bit complicated and
> confusing to users. I am wondering whether we can simplify it. A few
> questions below:
>
> - The KIP says that, with auto.offset.reset="closest", timestamp is used to
> find offset if truncation offset is unknown. It seems that if consumer
> knows the timestamp of the last message, then the consumer should also know
> the (offset, leaderEpoch) of the last message which can then be used for
> find the truncation offset. Can you explain why truncation offset is
> unknown in this case?
>
> - How does consumer differentiates between "Offset out of rnage (too low)"
> and "Offset out of range (unknown truncation offset)", i.e. the two columns
> in table provided in the KIP?
>
> - It is probably a typo. Maybe fix "This is not the last The" in the
> Proposed Section.
>
>
> Thanks,
> Dong
>
> On Mon, Jun 25, 2018 at 9:17 AM, Jason Gustafson 
> wrote:
>
> > Hey All,
> >
> > I wrote up a KIP to handle one more edge case in the replication protocol
> > and to support better handling of truncation in the consumer when unclean
> > leader election is enabled. Let me know what you think.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-320%3A
> > +Allow+fetchers+to+detect+and+handle+log+truncation
> >
> > Thanks to Anna Povzner and Dong Lin for initial feedback.
> >
> > Thanks,
> > Jason
> >
>

Re: SASL Unit test failing

2018-06-26 Thread Colin McCabe

On trunk, testMultipleServerMechanisms failed for me, as well as 
testAuthenticateCallbackHandlerMechanisms and testMechanismPluggability.

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMultipleServerMechanisms FAILED
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.kafka.common.network.NetworkTestUtils.waitForChannelReady(NetworkTestUtils.java:79)
at 
org.apache.kafka.common.network.NetworkTestUtils.checkClientConnection(NetworkTestUtils.java:52)
at 
org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest.testMultipleServerMechanisms(SaslAuthenticatorTest.java:281)

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testAuthenticateCallbackHandlerMechanisms FAILED
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.kafka.common.network.NetworkTestUtils.waitForChannelReady(NetworkTestUtils.java:79)
at 
org.apache.kafka.common.network.NetworkTestUtils.checkClientConnection(NetworkTestUtils.java:52)
at 
org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest.createAndCheckClientConnection(SaslAuthenticatorTest.java:1475)
at 
org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest.testAuthenticateCallbackHandlerMechanisms(SaslAuthenticatorTest.java:776)

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMechanismPluggability FAILED
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.kafka.common.network.NetworkTestUtils.waitForChannelReady(NetworkTestUtils.java:79)
at 
org.apache.kafka.common.network.NetworkTestUtils.checkClientConnection(NetworkTestUtils.java:52)
at 
org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest.createAndCheckClientConnection(SaslAuthenticatorTest.java:1475)
at 
org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest.testMechanismPluggability(SaslAuthenticatorTest.java:257)

best,
Colin

On Mon, Jun 25, 2018, at 18:02, Ted Yu wrote:
> I ran the test on Linux as well.
> 
> cat /etc/redhat-release
> CentOS Linux release 7.2.1511 (Core)
> 
> Java version: 1.8.0_161, vendor: Oracle Corporation
> Java home: /jdk1.8.0_161/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.10.0-327.28.3.el7.x86_64", arch: "amd64",
> family: "unix"
> 
> On Mon, Jun 25, 2018 at 5:42 PM, Ted Yu  wrote:
> 
> > Here was the command I used:
> >
> > ./gradlew -Dtest.single=SaslAuthenticatorTest clients:test
> >
> > On Mon, Jun 25, 2018 at 5:39 PM, Ahmed A  wrote:
> >
> >> I ran test with -i option as follows - "./gradlew  -i test".  The same set
> >> of three tests failed.
> >>
> >> My environment:
> >> $ java -version
> >> java version "1.8.0_121"
> >> Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
> >> Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
> >>
> >> $ cat /etc/redhat-release
> >> Red Hat Enterprise Linux Workstation release 7.3 (Maipo)
> >> $ uname -a
> >> Linux  ahmed  3.10.0-514.36.5.el7.x86_64 #1 SMP Thu Dec 28 21:42:18 EST
> >> 2017 x86_64 x86_64 x86_64 GNU/Linux
> >>
> >>
> >> Can you please let me know how I can run an individual unit test, what
> >> options do I provide?
> >>
> >>
> >> Thank you,
> >> Ahmed.
> >>
> >>
> >>
> >> On Mon, Jun 25, 2018 at 2:47 PM, Ted Yu  wrote:
> >>
> >> > I ran the test alone which passed.
> >> >
> >> > Can you include -i on the command line to see if there is some clue from
> >> > the output ?
> >> >
> >> > Here is my environment:
> >> >
> >> > Java version: 1.8.0_151, vendor: Oracle Corporation
> >> > Java home:
> >> > /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre
> >> > Default locale: en_US, platform encoding: UTF-8
> >> > OS name: "mac os x", version: "10.11.3", arch: "x86_64", family: "mac"
> >> >
> >> > FYI
> >> >
> >> > On Mon, Jun 25, 2018 at 12:59 PM, Ahmed A  wrote:
> >> >
> >> > > Hello,
> >> > >
> >> > > I did a fresh clone of the kafka src code, and the following SASL unit
> >> > > tests have been failing consistently:
> >> > > - testMechanismPluggability
> >> > > - testMechanismPluggability
> >> > > - testMultipleServerMechanisms
> >> > >
> >> > > All three tests have similar stack trace:
> >> > > at org.junit.Assert.assertTrue(Assert.java:52)
> >> > > at
> >> > > org.apache.kafka.common.network.NetworkTestUtils.waitForChannelReady(
> >> > > NetworkTestUtils.java:79)
> >> > > at
> >> > > org.apache.kafka.common.network.NetworkTestUtils.checkClient
> >> Connection(
> >> > >

[VOTE] KIP-324: Add method to get metrics() in AdminClient

Hi All,

I am starting a vote on this KIP:

https://cwiki.apache.org/confluence/x/lQg0BQ

Thanks,
Yishun

[jira] [Created] (KAFKA-7101) Session Window store should set topic policy `compact,cleanup`

2018-06-26 Thread Matthias J. Sax (JIRA)

Matthias J. Sax created KAFKA-7101:
--

 Summary: Session Window store should set topic policy 
`compact,cleanup`
 Key: KAFKA-7101
 URL: https://issues.apache.org/jira/browse/KAFKA-7101
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Affects Versions: 0.10.2.2, 2.0.0, 0.11.0.3, 1.0.2, 1.1.1
Reporter: Matthias J. Sax
Assignee: Guozhang Wang


With 
[KIP-71|https://cwiki.apache.org/confluence/display/KAFKA/KIP-71%3A+Enable+log+compaction+and+deletion+to+co-exist]
 (0.10.1.0) topic config `compact,delete` was introduce to apply to windowed 
store changelog topics in Kafka Streams. Later (0.10.2.0), session windows got 
added in 
[KIP-94|https://cwiki.apache.org/confluence/display/KAFKA/KIP-94+Session+Windows].
 However, session windows do not use `compact,delete` at the moment. This 
result is the same issue window stores face before KIP-71. Thus, we should 
enable `compact,delete` for session window changelog topics, too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[DISCUSS] KIP-325: Extend Consumer Group Command to Show Beginning Offsets

2018-06-26 Thread Vahid S Hashemian

Hi everyone,

I have created a trivial KIP to improve the offset reporting of the 
consumer group command: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-325%3A+Extend+Consumer+Group+Command+to+Show+Beginning+Offsets
Looking forward to your feedback!

Thanks.
--Vahid

[DISCUSS] KIP-326: Schedulable KTable as Graph source

2018-06-26 Thread Flávio Stutz

Hey, guys, I've just created a new KIP about creating a new DSL graph
source for realtime partitioned consolidations.

We have faced the following scenario/problem in a lot of situations with
KStreams:
- Huge incoming data being processed by numerous application instances
- Need to aggregate different fields whose records span all topic
partitions (something like “total amount spent by people aged > 30 yrs”
when processing a topic partitioned by userid).

The challenge here is to manage this kind of situation without any
bottlenecks. We don't need the “global aggregation” to be processed at each
incoming message. On a scenario of 500 instances, each handling 1k
messages/s, any single point of aggregation (single partitioned topics,
global tables or external databases) would create a bottleneck of 500k
messages/s for single threaded/CPU elements.

For this scenario, it is possible to store the partial aggregations on
local stores and, from time to time, query those states and aggregate them
as a single value, avoiding bottlenecks. This is a way to create a "timed
aggregation barrier”.

If we leverage this kind of built-in feature we could greatly enhance the
ability of KStreams to better handle the CAP Theorem characteristics, so
that one could choose to have Consistency over Availability when needed.

We started this discussion with Matthias J. Sax here:
https://issues.apache.org/jira/browse/KAFKA-6953

If you want to see more, go to KIP-326 at:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-326%3A+Schedulable+KTable+as+Graph+source

-Flávio Stutz

Re: [DISCUSSION] KIP-324: Add method to get metrics() in AdminClient

Sure, that sounds good. - Yishun

On Tue, Jun 26, 2018 at 11:19 AM, Guozhang Wang  wrote:

> Thanks Yishun.
>
> I think this should be a straight-forward one as "metrics()" is simply an
> overlooked API that AdminClient should have since day one, as other
> clients. We can go directly into the voting process.
>
> Guozhang
>
>
> On Tue, Jun 26, 2018 at 11:15 AM, Yishun Guan  wrote:
>
> > Hi All,
> >
> > I created a KIP to extract metrics() from AdminClient, here is the link:
> >
> > https://cwiki.apache.org/confluence/x/lQg0BQ
> >
> > Let me know what you think. Thanks @guozhangwang and @mjsax for guiding
> me.
> >
> > Best,
> > Yishun
> >
>
>
>
> --
> -- Guozhang
>

Re: [DISCUSSION] KIP-324: Add method to get metrics() in AdminClient

2018-06-26 Thread Guozhang Wang

Thanks Yishun.

I think this should be a straight-forward one as "metrics()" is simply an
overlooked API that AdminClient should have since day one, as other
clients. We can go directly into the voting process.

Guozhang

On Tue, Jun 26, 2018 at 11:15 AM, Yishun Guan  wrote:

> Hi All,
>
> I created a KIP to extract metrics() from AdminClient, here is the link:
>
> https://cwiki.apache.org/confluence/x/lQg0BQ
>
> Let me know what you think. Thanks @guozhangwang and @mjsax for guiding me.
>
> Best,
> Yishun
>

-- 
-- Guozhang

[DISCUSSION] KIP-324: Add method to get metrics() in AdminClient

Hi All,

I created a KIP to extract metrics() from AdminClient, here is the link:

https://cwiki.apache.org/confluence/x/lQg0BQ

Let me know what you think. Thanks @guozhangwang and @mjsax for guiding me.

Best,
Yishun

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

2018-06-26 Thread Dong Lin

Hey Jason,

Thanks for the KIP! It is pretty useful.

At high level the new set of reset policies may be a bit complicated and
confusing to users. I am wondering whether we can simplify it. A few
questions below:

- The KIP says that, with auto.offset.reset="closest", timestamp is used to
find offset if truncation offset is unknown. It seems that if consumer
knows the timestamp of the last message, then the consumer should also know
the (offset, leaderEpoch) of the last message which can then be used for
find the truncation offset. Can you explain why truncation offset is
unknown in this case?

- How does consumer differentiates between "Offset out of rnage (too low)"
and "Offset out of range (unknown truncation offset)", i.e. the two columns
in table provided in the KIP?

- It is probably a typo. Maybe fix "This is not the last The" in the
Proposed Section.


Thanks,
Dong

On Mon, Jun 25, 2018 at 9:17 AM, Jason Gustafson  wrote:

> Hey All,
>
> I wrote up a KIP to handle one more edge case in the replication protocol
> and to support better handling of truncation in the consumer when unclean
> leader election is enabled. Let me know what you think.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-320%3A
> +Allow+fetchers+to+detect+and+handle+log+truncation
>
> Thanks to Anna Povzner and Dong Lin for initial feedback.
>
> Thanks,
> Jason
>

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

2018-06-26 Thread Harsha

Thanks for the pointer. Will take a look might suit our requirements better.

Thanks,
Harsha

On Mon, Jun 25th, 2018 at 2:52 PM, Lucas Wang  wrote:

> 
> 
> 
> Hi Harsha,
> 
> If I understand correctly, the replication quota mechanism proposed in
> KIP-73 can be helpful in that scenario.
> Have you tried it out?
> 
> Thanks,
> Lucas
> 
> 
> 
> On Sun, Jun 24, 2018 at 8:28 AM, Harsha < ka...@harsha.io > wrote:
> 
> > Hi Lucas,
> > One more question, any thoughts on making this configurable
> > and also allowing subset of data requests to be prioritized. For example
> 
> > ,we notice in our cluster when we take out a broker and bring new one it
> 
> > will try to become follower and have lot of fetch requests to other
> leaders
> > in clusters. This will negatively effect the application/client
> requests.
> > We are also exploring the similar solution to de-prioritize if a new
> > replica comes in for fetch requests, we are ok with the replica to be
> > taking time but the leaders should prioritize the client requests.
> >
> >
> > Thanks,
> > Harsha
> >
> > On Fri, Jun 22nd, 2018 at 11:35 AM Lucas Wang wrote:
> >
> > >
> > >
> > >
> > > Hi Eno,
> > >
> > > Sorry for the delayed response.
> > > - I haven't implemented the feature yet, so no experimental results so
> 
> > > far.
> > > And I plan to test in out in the following days.
> > >
> > > - You are absolutely right that the priority queue does not completely
> 
> > > prevent
> > > data requests being processed ahead of controller requests.
> > > That being said, I expect it to greatly mitigate the effect of stable
> > > metadata.
> > > In any case, I'll try it out and post the results when I have it.
> > >
> > > Regards,
> > > Lucas
> > >
> > > On Wed, Jun 20, 2018 at 5:44 AM, Eno Thereska < eno.there...@gmail.com
> >
> > > wrote:
> > >
> > > > Hi Lucas,
> > > >
> > > > Sorry for the delay, just had a look at this. A couple of questions:
> 
> > > > - did you notice any positive change after implementing this KIP?
> I'm
> > > > wondering if you have any experimental results that show the benefit
> of
> > > the
> > > > two queues.
> > > >
> > > > - priority is usually not sufficient in addressing the problem the
> KIP
> > > > identifies. Even with priority queues, you will sometimes (often?)
> have
> > > the
> > > > case that data plane requests will be ahead of the control plane
> > > requests.
> > > > This happens because the system might have already started
> processing
> > > the
> > > > data plane requests before the control plane ones arrived. So it
> would
> > > be
> > > > good to know what % of the problem this KIP addresses.
> > > >
> > > > Thanks
> > > > Eno
> > > >
> > >
> > >
> > > > On Fri, Jun 15, 2018 at 4:44 PM, Ted Yu < yuzhih...@gmail.com >
> wrote:
> > > >
> > > > > Change looks good.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Fri, Jun 15, 2018 at 8:42 AM, Lucas Wang < lucasatu...@gmail.com
> 
> > >
> > > > wrote:
> > > > >
> > > > > > Hi Ted,
> > > > > >
> > > > > > Thanks for the suggestion. I've updated the KIP. Please take
> > another
> > >
> > > > > look.
> > > > > >
> > > > > > Lucas
> > > > > >
> > > > > > On Thu, Jun 14, 2018 at 6:34 PM, Ted Yu < yuzhih...@gmail.com >
> > > wrote:
> > > > > >
> > > > > > > Currently in KafkaConfig.scala :
> > > > > > >
> > > > > > > val QueuedMaxRequests = 500
> > > > > > >
> > > > > > > It would be good if you can include the default value for this
> 
> > new
> > >
> > > > > config
> > > > > > > in the KIP.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Thu, Jun 14, 2018 at 4:28 PM, Lucas Wang <
> > lucasatu...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Ted, Dong
> > > > > > > >
> > > > > > > > I've updated the KIP by adding a new config, instead of
> reusing
> > > the
> > > > > > > > existing one.
> > > > > > > > Please take another look when you have time. Thanks a lot!
> > > > > > > >
> > > > > > > > Lucas
> > > > > > > >
> > > > > > > > On Thu, Jun 14, 2018 at 2:33 PM, Ted Yu < yuzhih...@gmail.com
> 
> > >
> > > > wrote:
> > > > > > > >
> > > > > > > > > bq. that's a waste of resource if control request rate is
> low
> > > > > > > > >
> > > > > > > > > I don't know if control request rate can get to 100,000,
> > > likely
> > > > > not.
> > > > > > > Then
> > > > > > > > > using the same bound as that for data requests seems high.
> 
> > > > > > > > >
> > > > > > > > > On Wed, Jun 13, 2018 at 10:13 PM, Lucas Wang <
> > > > > lucasatu...@gmail.com >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Ted,
> > > > > > > > > >
> > > > > > > > > > Thanks for taking a look at this KIP.
> > > > > > > > > > Let's say today the setting of "queued.max.requests" in
> > > > cluster A
> > > > > > is
> > > > > > > > > 1000,
> > > > > > > > > > while the setting in cluster B is 100,000.
> > > > > > > > > > The 100 times difference might have indicated that
> machines
> > > in
> > > > > > > cluster
> > > > > > > > B
> > > > > > > > > >

Re: [DISCUSS] KIP-308: Support dynamic update of max.connections.per.ip/max.connections.per.ip.overrides configs

2018-06-26 Thread Harsha

This is very useful. LGTM.

Thanks,
Harsha

On Mon, Jun 25th, 2018 at 10:20 AM, Dong Lin  wrote:

> 
> 
> 
> Hey Manikumar,
> 
> Thanks much for the KIP. It looks pretty good.
> 
> Thanks,
> Dong
> 
> On Thu, Jun 21, 2018 at 11:38 PM, Manikumar < manikumar.re...@gmail.com >
> wrote:
> 
> > Hi all,
> >
> > I have created a KIP to add support for dynamic update of
> > max.connections.per.ip/max.connections.per.ip.overrides configs
> >
> > * https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=85474993
> 
> > < https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=85474993
> 
> > >*
> >
> > Any feedback is appreciated.
> >
> > Thanks
> >
> 
> 
> 
>

[DISCUSS] KIP-322: Return new error code for DeleteTopics API when topic deletion disabled.

2018-06-26 Thread Manikumar

Hi all,

I have created a minor KIP to return new error code for DeleteTopics API
when topic deletion disabled.

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87295558

Please take a look.

Thanks,

Re: [VOTE] KIP-319: Replace numSegments to segmentInterval in Streams window configurations

2018-06-26 Thread Bill Bejeck

+1

On Mon, Jun 25, 2018 at 11:07 PM Matthias J. Sax 
wrote:

> +1 (binding)
>
> On 6/25/18 3:00 PM, Guozhang Wang wrote:
> > +1
> >
> > On Mon, Jun 25, 2018 at 2:58 PM, Ted Yu  wrote:
> >
> >> +1
> >>
> >> On Mon, Jun 25, 2018 at 2:56 PM, John Roesler 
> wrote:
> >>
> >>> Hello All,
> >>>
> >>> Thanks for the discussion on KIP-319. I'd now like to start the voting.
> >>>
> >>> As a reminder, KIP-319 proposes a fix to an issue I identified in
> >>> KAFKA-7080. Specifically, the issue is that we're creating
> >>> CachingWindowStore with the *number of segments* instead of the
> *segment
> >>> size*.
> >>>
> >>> Here's the jira: https://issues.apache.org/jira/browse/KAFKA-7080
> >>> Here's the KIP: https://cwiki.apache.org/confluence/x/mQU0BQ
> >>>
> >>> Additionally, here's a draft PR for clarity:
> >>> https://github.com/apache/kafka/pull/5257
> >>>
> >>> Thanks,
> >>> -John
> >>>
> >>
> >
> >
> >
>
>

[jira] [Resolved] (KAFKA-7100) kafka.tools.GetOffsetShell with enable Kerberos Security on Kafka1.0



 [ 
https://issues.apache.org/jira/browse/KAFKA-7100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-7100.
--
Resolution: Duplicate

This is being tracked in KAFKA-5235.

> kafka.tools.GetOffsetShell with enable Kerberos Security on Kafka1.0  
> -
>
> Key: KAFKA-7100
> URL: https://issues.apache.org/jira/browse/KAFKA-7100
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 1.0.1
>Reporter: abdullah toraman
>Priority: Major
>
> Hi All,
> I enabled the Kerberos Authentication on Kafka 1.0. When I try to 
> kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list : 
> --topic  --time -1, It hits error. Here are the Error,
> [kafka@KLBIRKFKI1 kafka_2.11-1.0.1]$ bin/kafka-run-class.sh 
> kafka.tools.GetOffsetShell --broker-list klbirkfki1:9092 --topic abdullah 
> --time -1
> [2018-06-26 12:59:00,058] WARN Fetching topic metadata with correlation id 0 
> for topics [Set(abdullah)] from broker [BrokerEndPoint(0,klbirkfki1,9092)] 
> failed (kafka.client.ClientUtils$)
> java.io.EOFException
>     at 
> org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:124)
>     at 
> kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:131)
>     at kafka.network.BlockingChannel.receive(BlockingChannel.scala:122)
>     at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:82)
>     at 
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79)
>     at kafka.producer.SyncProducer.send(SyncProducer.scala:124)
>     at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:63)
>     at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:98)
>     at kafka.tools.GetOffsetShell$.main(GetOffsetShell.scala:79)
>     at kafka.tools.GetOffsetShell.main(GetOffsetShell.scala)
> Exception in thread "main" kafka.common.KafkaException: fetching topic 
> metadata for topics [Set(abdullah)] from broker 
> [ArrayBuffer(BrokerEndPoint(0,klbirkfki1,9092))] failed
>     at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:77)
>     at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:98)
>     at kafka.tools.GetOffsetShell$.main(GetOffsetShell.scala:79)
>     at kafka.tools.GetOffsetShell.main(GetOffsetShell.scala)
> Caused by: java.io.EOFException
>     at 
> org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:124)
>     at 
> kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:131)
>     at kafka.network.BlockingChannel.receive(BlockingChannel.scala:122)
>     at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:82)
>     at 
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79)
>     at kafka.producer.SyncProducer.send(SyncProducer.scala:124)
>     at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:63)
>     ... 3 more
> [kafka@KLBIRKFKI1 kafka_2.11-1.0.1]$ bin/kafka-run-class.sh 
> kafka.tools.GetOffsetShell --broker-list klbirkfki1:9092 --topic abdullah 
> --time -1
> [2018-06-26 12:59:00,058] WARN Fetching topic metadata with correlation id 0 
> for topics [Set(abdullah)] from broker [BrokerEndPoint(0,klbirkfki1,9092)] 
> failed (kafka.client.ClientUtils$)
> java.io.EOFException
>     at 
> org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:124)
>     at 
> kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:131)
>     at kafka.network.BlockingChannel.receive(BlockingChannel.scala:122)
>     at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:82)
>     at 
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79)
>     at kafka.producer.SyncProducer.send(SyncProducer.scala:124)
>     at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:63)
>     at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:98)
>     at kafka.tools.GetOffsetShell$.main(GetOffsetShell.scala:79)
>     at kafka.tools.GetOffsetShell.main(GetOffsetShell.scala)
> Exception in thread "main" kafka.common.KafkaException: fetching topic 
> metadata for topics [Set(abdullah)] from broker 
> [ArrayBuffer(BrokerEndPoint(0,klbirkfki1,9092))] failed
>     at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:77)
>     at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:98)
>     at kafka.tools.GetOffsetShell$.main(GetOffsetShell.scala:79)
>     at kafka.tools.GetOffsetShell.main(GetOffsetShell.scala)
> Caused by: java.io.EOFException
>     at 
>

[jira] [Created] (KAFKA-7100) kafka.tools.GetOffsetShell with enable Kerberos Security on Kafka1.0

2018-06-26 Thread abdullah toraman (JIRA)

abdullah toraman created KAFKA-7100:
---

 Summary: kafka.tools.GetOffsetShell with enable Kerberos Security 
on Kafka1.0  
 Key: KAFKA-7100
 URL: https://issues.apache.org/jira/browse/KAFKA-7100
 Project: Kafka
  Issue Type: Bug
Affects Versions: 1.0.1
Reporter: abdullah toraman


Hi All,

I enabled the Kerberos Authentication on Kafka 1.0. When I try to 
kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list : --topic 
 --time -1, It hits error. Here are the Error,

[kafka@KLBIRKFKI1 kafka_2.11-1.0.1]$ bin/kafka-run-class.sh 
kafka.tools.GetOffsetShell --broker-list klbirkfki1:9092 --topic abdullah 
--time -1

[2018-06-26 12:59:00,058] WARN Fetching topic metadata with correlation id 0 
for topics [Set(abdullah)] from broker [BrokerEndPoint(0,klbirkfki1,9092)] 
failed (kafka.client.ClientUtils$)

java.io.EOFException

    at 
org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:124)

    at 
kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:131)

    at kafka.network.BlockingChannel.receive(BlockingChannel.scala:122)

    at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:82)

    at 
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79)

    at kafka.producer.SyncProducer.send(SyncProducer.scala:124)

    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:63)

    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:98)

    at kafka.tools.GetOffsetShell$.main(GetOffsetShell.scala:79)

    at kafka.tools.GetOffsetShell.main(GetOffsetShell.scala)

Exception in thread "main" kafka.common.KafkaException: fetching topic metadata 
for topics [Set(abdullah)] from broker 
[ArrayBuffer(BrokerEndPoint(0,klbirkfki1,9092))] failed

    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:77)

    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:98)

    at kafka.tools.GetOffsetShell$.main(GetOffsetShell.scala:79)

    at kafka.tools.GetOffsetShell.main(GetOffsetShell.scala)

Caused by: java.io.EOFException

    at 
org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:124)

    at 
kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:131)

    at kafka.network.BlockingChannel.receive(BlockingChannel.scala:122)

    at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:82)

    at 
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79)

    at kafka.producer.SyncProducer.send(SyncProducer.scala:124)

    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:63)

    ... 3 more

[kafka@KLBIRKFKI1 kafka_2.11-1.0.1]$ bin/kafka-run-class.sh 
kafka.tools.GetOffsetShell --broker-list klbirkfki1:9092 --topic abdullah 
--time -1

[2018-06-26 12:59:00,058] WARN Fetching topic metadata with correlation id 0 
for topics [Set(abdullah)] from broker [BrokerEndPoint(0,klbirkfki1,9092)] 
failed (kafka.client.ClientUtils$)

java.io.EOFException

    at 
org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:124)

    at 
kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:131)

    at kafka.network.BlockingChannel.receive(BlockingChannel.scala:122)

    at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:82)

    at 
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79)

    at kafka.producer.SyncProducer.send(SyncProducer.scala:124)

    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:63)

    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:98)

    at kafka.tools.GetOffsetShell$.main(GetOffsetShell.scala:79)

    at kafka.tools.GetOffsetShell.main(GetOffsetShell.scala)

Exception in thread "main" kafka.common.KafkaException: fetching topic metadata 
for topics [Set(abdullah)] from broker 
[ArrayBuffer(BrokerEndPoint(0,klbirkfki1,9092))] failed

    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:77)

    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:98)

    at kafka.tools.GetOffsetShell$.main(GetOffsetShell.scala:79)

    at kafka.tools.GetOffsetShell.main(GetOffsetShell.scala)

Caused by: java.io.EOFException

    at 
org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:124)

    at 
kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:131)

    at kafka.network.BlockingChannel.receive(BlockingChannel.scala:122)

    at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:82)

    at

[jira] [Created] (KAFKA-7099) KafkaLog4jAppender - not sending any message with level DEBUG

2018-06-26 Thread Vincent Lebreil (JIRA)

Vincent Lebreil created KAFKA-7099:
--

 Summary: KafkaLog4jAppender - not sending any message with level 
DEBUG
 Key: KAFKA-7099
 URL: https://issues.apache.org/jira/browse/KAFKA-7099
 Project: Kafka
  Issue Type: Bug
  Components: log
Affects Versions: 0.10.2.0
Reporter: Vincent Lebreil


KafkaLog4jAppender can be stuck if it is defined at root category with level 
DEBUG

{{log4j.rootLogger=DEBUG, kafka}}

{{log4j.appender.kafka=org.apache.kafka.log4jappender.KafkaLog4jAppender}}
{quote}DEBUG org.apache.kafka.clients.producer.KafkaProducer:131 - Exception 
occurred during message send:

org.apache.kafka.common.errors.TimeoutException: Failed to update metadata 
after 6 ms.
{quote}
KafkaLog4jAppender is using a KafkaProducer using itself Log4j with messages at 
levels TRACE and DEBUG. The appender used in this case is also the 
KafkaLog4jAppender.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-5079) ProducerBounceTest fails occasionally with a SocketTimeoutException



 [ 
https://issues.apache.org/jira/browse/KAFKA-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-5079.
--
Resolution: Fixed

ProducerBounceTest is removed as part old consumer changes.

> ProducerBounceTest fails occasionally with a SocketTimeoutException
> ---
>
> Key: KAFKA-5079
> URL: https://issues.apache.org/jira/browse/KAFKA-5079
> Project: Kafka
>  Issue Type: Bug
>Reporter: Apurva Mehta
>Priority: Major
>
> {noformat}
> java.net.SocketTimeoutException
>   at 
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:229)
>   at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
>   at 
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
>   at 
> org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:85)
>   at 
> kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:129)
>   at kafka.network.BlockingChannel.receive(BlockingChannel.scala:120)
>   at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:100)
>   at 
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:84)
>   at 
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:133)
>   at 
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:133)
>   at 
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:133)
>   at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:32)
>   at 
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:132)
>   at 
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:132)
>   at 
> kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:132)
>   at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:32)
>   at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:131)
>   at 
> kafka.api.ProducerBounceTest$$anonfun$2.apply(ProducerBounceTest.scala:116)
>   at 
> kafka.api.ProducerBounceTest$$anonfun$2.apply(ProducerBounceTest.scala:113)
> {noformat}
> This is expected occasionally, since the ports are preallocated and the 
> brokers are bounced in quick succession. Here is the relevant comment from 
> the code: 
> {noformat}
>   // This is the one of the few tests we currently allow to preallocate 
> ports, despite the fact that this can result in transient
>   // failures due to ports getting reused. We can't use random ports because 
> of bad behavior that can result from bouncing
>   // brokers too quickly when they get new, random ports. If we're not 
> careful, the client can end up in a situation
>   // where metadata is not refreshed quickly enough, and by the time it's 
> actually trying to, all the servers have
>   // been bounced and have new addresses. None of the bootstrap nodes or 
> current metadata can get them connected to a
>   // running server.
>   //
>   // Since such quick rotation of servers is incredibly unrealistic, we allow 
> this one test to preallocate ports, leaving
>   // a small risk of hitting errors due to port conflicts. Hopefully this is 
> infrequent enough to not cause problems.
> {noformat}
> We should try to look into handling this exception better so that the test 
> doesn't fail occasionally. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-5080) Convert ProducerBounceTest to use the new KafkaConsumer



 [ 
https://issues.apache.org/jira/browse/KAFKA-5080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-5080.
--
Resolution: Fixed

ProducerBounceTest is removed as part of old consumer changes.

> Convert ProducerBounceTest to use the new KafkaConsumer
> ---
>
> Key: KAFKA-5080
> URL: https://issues.apache.org/jira/browse/KAFKA-5080
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
>Priority: Major
>
> In KAFKA-5079, the `SocketTimeoutException` indicates that the consumer is 
> stuck waiting for a response from a particular broker, even after the broker 
> has been bounced. Since a given broker in that test always uses the same port 
> is is possible that this is a symptom of a bug in the SimpleConsumer where it 
> doesn't detect the bounce (and hence disconnect), causing it to time out. 
> We should use the new consumer to rule out a client bug, or fix it if it also 
> exists in the new consumer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-2488) System tests: updated console_consumer.py to support new consumer



 [ 
https://issues.apache.org/jira/browse/KAFKA-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2488.
--
Resolution: Fixed

Support was added in > 0.10.1 Kafka versions.

> System tests: updated console_consumer.py to support new consumer
> -
>
> Key: KAFKA-2488
> URL: https://issues.apache.org/jira/browse/KAFKA-2488
> Project: Kafka
>  Issue Type: Bug
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
>Priority: Major
>
> Console consumer now supports new consumer
> Update console_consumer.py to allow this as well



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-2215) Improve Randomness for ConsoleConsumer



 [ 
https://issues.apache.org/jira/browse/KAFKA-2215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2215.
--
Resolution: Not A Problem

Closing inactive issue. Also the default for console consumer's 
enable.auto.commit is set to false for auto-generated group Ids. 

> Improve Randomness for ConsoleConsumer
> --
>
> Key: KAFKA-2215
> URL: https://issues.apache.org/jira/browse/KAFKA-2215
> Project: Kafka
>  Issue Type: Bug
>Reporter: Fabian Lange
>Priority: Major
>
> Right now the console consumer does a new Random().nextInt(100_000)
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L123
> I would propose to use UUID.randomUUID().toString() instead.
> I know this is quite edgy, but Random has shown its quirks from time to time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : kafka-trunk-jdk8 #2768