date:20180828

See 


Changes:

[jason] KAFKA-7128; Follower has to catch up to offset within current leader

--
[...truncated 581.88 KB...]
kafka.api.SaslSslAdminClientIntegrationTest > testMinimumRequestTimeouts PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testForceClose STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testForceClose PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testListNodes STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testListNodes PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testDelayedClose STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testDelayedClose PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testCreateDeleteTopics STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testCreateDeleteTopics PASSED

kafka.api.SaslSslAdminClientIntegrationTest > 
testOffsetsForTimesAfterDeleteRecords STARTED

kafka.api.SaslSslAdminClientIntegrationTest > 
testOffsetsForTimesAfterDeleteRecords PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testDeleteRecordsWithException 
STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testDeleteRecordsWithException 
PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testDescribeLogDirs STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testDescribeLogDirs PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testAlterReplicaLogDirs STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testAlterReplicaLogDirs PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testLogStartOffsetCheckpoint 
STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testLogStartOffsetCheckpoint 
PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testDescribeCluster STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testDescribeCluster PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testCreatePartitions STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testCreatePartitions PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testDescribeNonExistingTopic 
STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testDescribeNonExistingTopic 
PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testMetadataRefresh STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testMetadataRefresh PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testDescribeAndAlterConfigs 
STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testDescribeAndAlterConfigs PASSED

kafka.api.SaslSslAdminClientIntegrationTest > 
testLogStartOffsetAfterDeleteRecords STARTED

kafka.api.SaslSslAdminClientIntegrationTest > 
testLogStartOffsetAfterDeleteRecords PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testSeekAfterDeleteRecords STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testSeekAfterDeleteRecords PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testCallInFlightTimeouts STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testCallInFlightTimeouts PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testDescribeConfigsForTopic 
STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testDescribeConfigsForTopic PASSED

kafka.api.SaslSslAdminClientIntegrationTest > testConsumerGroups STARTED

kafka.api.SaslSslAdminClientIntegrationTest > testConsumerGroups PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testAcls STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testAcls PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testTwoConsumersWithDifferentSaslCredentials PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithoutDescribeAclViaSubscribe PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testProduceConsumeWithPrefixedAcls STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testProduceConsumeWithPrefixedAcls PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaAssign PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testProduceConsumeTopicAutoCreateTopicCreateAcl STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testProduceConsumeTopicAutoCreateTopicCreateAcl PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testProduceConsumeWithWildcardAcls STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testProduceConsumeWithWildcardAcls PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > 
testNoConsumeWithDescribeAclViaSubscribe STARTED

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

2018-08-28 Thread Randall Hauch

Do you think we should support name-value pairs, too?

On Tue, Aug 28, 2018 at 6:41 PM Magesh Nandakumar 
wrote:

> Randall,
>
> Thanks a lot for the KIP. I think this would be a great addition for many
> source connectors.
> One clarification I had was regarding the topic settings that can be
> configured. Is it limited to the setting exposed in the TopicSettings
> interface?
>
> Thanks
> Magesh
>
> On Tue, Aug 21, 2018 at 7:59 PM Randall Hauch  wrote:
>
> > Okay, after much delay let's try this again for AK 2.1. Has anyone found
> > any concerns? Stephane suggested that we allow updating topic
> > configurations (everything but partition count). I'm unconvinced that
> it's
> > worth the additional complexity in the implementation and the
> documentation
> > to explain the behavior. Changing several of the topic-specific
> > configurations have significant impact on broker behavior /
> functionality,
> > so IMO we need to proceed more cautiously.
> >
> > Stephane, do you have a particular use case in mind for updating topic
> > configurations on an existing topic?
> >
> > Randall
> >
> >
> > On Fri, Jan 26, 2018 at 4:20 PM Randall Hauch  wrote:
> >
> > > The KIP deadline for 1.1 has already passed, but I'd like to restart
> this
> > > discussion so that we make the next release. I've not yet addressed the
> > > previous comment about *existing* topics, but I'll try to do that over
> > the
> > > next few weeks. Any other comments/suggestions/questions?
> > >
> > > Best regards,
> > >
> > > Randall
> > >
> > > On Thu, Oct 5, 2017 at 12:13 AM, Randall Hauch 
> wrote:
> > >
> > >> Oops. Yes, I meant “replication factor”.
> > >>
> > >> > On Oct 4, 2017, at 7:18 PM, Ted Yu  wrote:
> > >> >
> > >> > Randall:
> > >> > bq. AdminClient currently allows changing the replication factory.
> > >> >
> > >> > By 'replication factory' did you mean 'replication factor' ?
> > >> >
> > >> > Cheers
> > >> >
> > >> >> On Wed, Oct 4, 2017 at 9:58 AM, Randall Hauch 
> > >> wrote:
> > >> >>
> > >> >> Currently the KIP's scope is only topics that don't yet exist, and
> we
> > >> have
> > >> >> to cognizant of race conditions between tasks with the same
> > connector.
> > >> I
> > >> >> think it is worthwhile to consider whether the KIP's scope should
> > >> expand to
> > >> >> also address *existing* partitions, though it may not be
> appropriate
> > to
> > >> >> have as much control when changing the topic settings for an
> existing
> > >> >> topic. For example, changing the number of partitions (which the
> KIP
> > >> >> considers a "topic-specific setting" even though technically it is
> > not)
> > >> >> shouldn't be done blindly due to the partitioning impacts, and IIRC
> > you
> > >> >> can't reduce them (which we could verify before applying). Also, I
> > >> don't
> > >> >> think the AdminClient currently allows changing the replication
> > >> factory. I
> > >> >> think changing the topic configs is less problematic both from what
> > >> makes
> > >> >> sense for connectors to verify/change and from what the AdminClient
> > >> >> supports.
> > >> >>
> > >> >> Even if we decide that it's not appropriate to change the settings
> on
> > >> an
> > >> >> existing topic, I do think it's advantageous to at least notify the
> > >> >> connector (or task) prior to the first record sent to a given topic
> > so
> > >> that
> > >> >> the connector can fail or issue a warning if it doesn't meet its
> > >> >> requirements.
> > >> >>
> > >> >> Best regards,
> > >> >>
> > >> >> Randall
> > >> >>
> > >> >> On Wed, Oct 4, 2017 at 12:52 AM, Stephane Maarek <
> > >> >> steph...@simplemachines.com.au> wrote:
> > >> >>
> > >> >>> Hi Randall,
> > >> >>>
> > >> >>> Thanks for the KIP. I like it
> > >> >>> What happens when the target topic is already created but the
> > configs
> > >> do
> > >> >>> not match?
> > >> >>> i.e. wrong RF, num partitions, or missing / additional configs?
> Will
> > >> you
> > >> >>> attempt to apply the necessary changes or throw an error?
> > >> >>>
> > >> >>> Thanks!
> > >> >>> Stephane
> > >> >>>
> > >> >>>
> > >> >>> On 24/5/17, 5:59 am, "Mathieu Fenniak" <
> > mathieu.fenn...@replicon.com
> > >> >
> > >> >>> wrote:
> > >> >>>
> > >> >>>Ah, yes, I see you a highlighted part that should've made this
> > >> clear
> > >> >>>to me the first read. :-)  Much clearer now!
> > >> >>>
> > >> >>>By the way, enjoyed your Debezium talk in NYC.
> > >> >>>
> > >> >>>Looking forward to this Kafka Connect change; it will allow me
> to
> > >> >>>remove a post-deployment tool that I hacked together for the
> > >> purpose
> > >> >>>of ensuring auto-created topics have the right config.
> > >> >>>
> > >> >>>Mathieu
> > >> >>>
> > >> >>>
> > >> >>>On Tue, May 23, 2017 at 11:38 AM, Randall Hauch <
> > rha...@gmail.com>
> > >> >>> wrote:
> > >>  Thanks for the quick feedback, Mathieu. Yes, the first
> > >> >> configuration
> > >> >>> rule
> > >>  whose regex matches will be applied, and no other

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

2018-08-28 Thread Magesh Nandakumar

Randall,

Thanks a lot for the KIP. I think this would be a great addition for many
source connectors.
One clarification I had was regarding the topic settings that can be
configured. Is it limited to the setting exposed in the TopicSettings
interface?

Thanks
Magesh

On Tue, Aug 21, 2018 at 7:59 PM Randall Hauch  wrote:

> Okay, after much delay let's try this again for AK 2.1. Has anyone found
> any concerns? Stephane suggested that we allow updating topic
> configurations (everything but partition count). I'm unconvinced that it's
> worth the additional complexity in the implementation and the documentation
> to explain the behavior. Changing several of the topic-specific
> configurations have significant impact on broker behavior / functionality,
> so IMO we need to proceed more cautiously.
>
> Stephane, do you have a particular use case in mind for updating topic
> configurations on an existing topic?
>
> Randall
>
>
> On Fri, Jan 26, 2018 at 4:20 PM Randall Hauch  wrote:
>
> > The KIP deadline for 1.1 has already passed, but I'd like to restart this
> > discussion so that we make the next release. I've not yet addressed the
> > previous comment about *existing* topics, but I'll try to do that over
> the
> > next few weeks. Any other comments/suggestions/questions?
> >
> > Best regards,
> >
> > Randall
> >
> > On Thu, Oct 5, 2017 at 12:13 AM, Randall Hauch  wrote:
> >
> >> Oops. Yes, I meant “replication factor”.
> >>
> >> > On Oct 4, 2017, at 7:18 PM, Ted Yu  wrote:
> >> >
> >> > Randall:
> >> > bq. AdminClient currently allows changing the replication factory.
> >> >
> >> > By 'replication factory' did you mean 'replication factor' ?
> >> >
> >> > Cheers
> >> >
> >> >> On Wed, Oct 4, 2017 at 9:58 AM, Randall Hauch 
> >> wrote:
> >> >>
> >> >> Currently the KIP's scope is only topics that don't yet exist, and we
> >> have
> >> >> to cognizant of race conditions between tasks with the same
> connector.
> >> I
> >> >> think it is worthwhile to consider whether the KIP's scope should
> >> expand to
> >> >> also address *existing* partitions, though it may not be appropriate
> to
> >> >> have as much control when changing the topic settings for an existing
> >> >> topic. For example, changing the number of partitions (which the KIP
> >> >> considers a "topic-specific setting" even though technically it is
> not)
> >> >> shouldn't be done blindly due to the partitioning impacts, and IIRC
> you
> >> >> can't reduce them (which we could verify before applying). Also, I
> >> don't
> >> >> think the AdminClient currently allows changing the replication
> >> factory. I
> >> >> think changing the topic configs is less problematic both from what
> >> makes
> >> >> sense for connectors to verify/change and from what the AdminClient
> >> >> supports.
> >> >>
> >> >> Even if we decide that it's not appropriate to change the settings on
> >> an
> >> >> existing topic, I do think it's advantageous to at least notify the
> >> >> connector (or task) prior to the first record sent to a given topic
> so
> >> that
> >> >> the connector can fail or issue a warning if it doesn't meet its
> >> >> requirements.
> >> >>
> >> >> Best regards,
> >> >>
> >> >> Randall
> >> >>
> >> >> On Wed, Oct 4, 2017 at 12:52 AM, Stephane Maarek <
> >> >> steph...@simplemachines.com.au> wrote:
> >> >>
> >> >>> Hi Randall,
> >> >>>
> >> >>> Thanks for the KIP. I like it
> >> >>> What happens when the target topic is already created but the
> configs
> >> do
> >> >>> not match?
> >> >>> i.e. wrong RF, num partitions, or missing / additional configs? Will
> >> you
> >> >>> attempt to apply the necessary changes or throw an error?
> >> >>>
> >> >>> Thanks!
> >> >>> Stephane
> >> >>>
> >> >>>
> >> >>> On 24/5/17, 5:59 am, "Mathieu Fenniak" <
> mathieu.fenn...@replicon.com
> >> >
> >> >>> wrote:
> >> >>>
> >> >>>Ah, yes, I see you a highlighted part that should've made this
> >> clear
> >> >>>to me the first read. :-)  Much clearer now!
> >> >>>
> >> >>>By the way, enjoyed your Debezium talk in NYC.
> >> >>>
> >> >>>Looking forward to this Kafka Connect change; it will allow me to
> >> >>>remove a post-deployment tool that I hacked together for the
> >> purpose
> >> >>>of ensuring auto-created topics have the right config.
> >> >>>
> >> >>>Mathieu
> >> >>>
> >> >>>
> >> >>>On Tue, May 23, 2017 at 11:38 AM, Randall Hauch <
> rha...@gmail.com>
> >> >>> wrote:
> >>  Thanks for the quick feedback, Mathieu. Yes, the first
> >> >> configuration
> >> >>> rule
> >>  whose regex matches will be applied, and no other rules will be
> >> >>> used. I've
> >>  updated the KIP to try to make this more clear, but let me know if
> >> >>> it's
> >>  still not clear.
> >> 
> >>  Best regards,
> >> 
> >>  Randall
> >> 
> >>  On Tue, May 23, 2017 at 10:07 AM, Mathieu Fenniak <
> >>  mathieu.fenn...@replicon.com> wrote:
> >> 
> >> > Hi Randall,
> >> >
> >> > Awesome, very much

Jenkins build is back to normal : kafka-trunk-jdk10 #441

See

Re: [VOTE] KIP-275 - Indicate "isClosing" in the SinkTaskContext

2018-08-28 Thread Matt Farmer

Given that voting and discussion have stalled out it seems like this is a
thing that folks aren't particularly interested in. I'll be moving the KIP
status to abandoned unless I hear an objection in the next day or so. :)

On Thu, May 31, 2018 at 12:39 PM Matt Farmer  wrote:

> Bumping this again as it's been languishing for a few weeks. Would love to
> get further feedback (or know for sure that this won't happen).
>
> On Mon, May 14, 2018 at 3:48 PM, Matt Farmer  wrote:
>
>> Bumping this thread.
>>
>> For anyone who needs a refresher the discussion thread is here:
>> http://mail-archives.apache.org/mod_mbox/kafka-dev/201803.mbox/%3CCAM5dya9x---9M3uEf_wrJL5dw%2B6HLV4%3D5PfKKSTPE1vOHEWC_g%40mail.gmail.com%3E
>>
>> And there's a work in progress PR open here:
>> https://github.com/apache/kafka/pull/5002
>>
>> Thanks!
>>
>> On Wed, Apr 25, 2018 at 1:04 PM, Matt Farmer  wrote:
>>
>>> Bump!
>>>
>>> We're currently at 1 non-binding +1.
>>>
>>> Still soliciting votes here. =)
>>>
>>> On Wed, Apr 18, 2018 at 3:41 PM, Ted Yu  wrote:
>>>
 +1

 On Wed, Apr 18, 2018 at 12:40 PM, Matt Farmer  wrote:

 > Good afternoon/evening/morning all:
 >
 > I'd like to start voting on KIP-275: Indicate "isClosing" in the
 > SinkTaskContext
 >
 https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75977607
 >
 > I'm going to start preparing the patch we've been using internally
 for PR
 > and get it up for review later this week.
 >
 > Thanks!
 > Matt
 >

>>>
>>>
>>
>

Build failed in Jenkins: kafka-1.1-jdk7 #186

See 


Changes:

[jason] KAFKA-7164; Follower should truncate after every missed leader epoch

[jason] KAFKA-7128; Follower has to catch up to offset within current leader

--
[...truncated 1.93 MB...]
org.apache.kafka.streams.kstream.internals.KTableAggregateTest > 
testRemoveOldBeforeAddNew STARTED

org.apache.kafka.streams.kstream.internals.KTableAggregateTest > 
testRemoveOldBeforeAddNew PASSED

org.apache.kafka.streams.kstream.internals.KTableAggregateTest > 
testCountCoalesced STARTED

org.apache.kafka.streams.kstream.internals.KTableAggregateTest > 
testCountCoalesced PASSED

org.apache.kafka.streams.kstream.internals.KTableForeachTest > testForeach 
STARTED

org.apache.kafka.streams.kstream.internals.KTableForeachTest > testForeach 
PASSED

org.apache.kafka.streams.kstream.internals.KTableForeachTest > testTypeVariance 
STARTED

org.apache.kafka.streams.kstream.internals.KTableForeachTest > testTypeVariance 
PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
retentionTimeShouldBeGapIfGapIsLargerThanDefaultRetentionTime STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
retentionTimeShouldBeGapIfGapIsLargerThanDefaultRetentionTime PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > shouldSetWindowGap STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > shouldSetWindowGap PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldBeEqualWhenGapAndMaintainMsAreTheSame STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldBeEqualWhenGapAndMaintainMsAreTheSame PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
retentionTimeMustNotBeNegative STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
retentionTimeMustNotBeNegative PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldNotBeEqualWhenMaintainMsDifferent STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldNotBeEqualWhenMaintainMsDifferent PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > windowSizeMustNotBeZero 
STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > windowSizeMustNotBeZero 
PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
windowSizeMustNotBeNegative STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
windowSizeMustNotBeNegative PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldSetWindowRetentionTime STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldSetWindowRetentionTime PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldNotBeEqualWhenGapIsDifferent STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldNotBeEqualWhenGapIsDifferent PASSED

org.apache.kafka.streams.kstream.WindowTest > 
shouldThrowIfEndIsSmallerThanStart STARTED

org.apache.kafka.streams.kstream.WindowTest > 
shouldThrowIfEndIsSmallerThanStart PASSED

org.apache.kafka.streams.kstream.WindowTest > 
shouldNotBeEqualIfDifferentWindowType STARTED

org.apache.kafka.streams.kstream.WindowTest > 
shouldNotBeEqualIfDifferentWindowType PASSED

org.apache.kafka.streams.kstream.WindowTest > shouldBeEqualIfStartAndEndSame 
STARTED

org.apache.kafka.streams.kstream.WindowTest > shouldBeEqualIfStartAndEndSame 
PASSED

org.apache.kafka.streams.kstream.WindowTest > shouldNotBeEqualIfNull STARTED

org.apache.kafka.streams.kstream.WindowTest > shouldNotBeEqualIfNull PASSED

org.apache.kafka.streams.kstream.WindowTest > shouldThrowIfStartIsNegative 
STARTED

org.apache.kafka.streams.kstream.WindowTest > shouldThrowIfStartIsNegative 
PASSED

org.apache.kafka.streams.kstream.WindowTest > 
shouldNotBeEqualIfStartOrEndIsDifferent STARTED

org.apache.kafka.streams.kstream.WindowTest > 
shouldNotBeEqualIfStartOrEndIsDifferent PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldAddRegexTopicToLatestAutoOffsetResetList STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldAddRegexTopicToLatestAutoOffsetResetList PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldMapStateStoresToCorrectSourceTopics STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldMapStateStoresToCorrectSourceTopics PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldAddTableToEarliestAutoOffsetResetList STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldAddTableToEarliestAutoOffsetResetList PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldAddTimestampExtractorToTablePerSource STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldAddTimestampExtractorToTablePerSource PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldBuildGlobalTopologyWithAllGlobalTables STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldBuildGlobalTopologyWithAllGlobalTables PASSED

[jira] [Created] (KAFKA-7353) Connect logs 'this' for anonymous inner classes

2018-08-28 Thread Kevin Lafferty (JIRA)

Kevin Lafferty created KAFKA-7353:
-

 Summary: Connect logs 'this' for anonymous inner classes
 Key: KAFKA-7353
 URL: https://issues.apache.org/jira/browse/KAFKA-7353
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 2.0.0, 1.1.1, 1.0.2
Reporter: Kevin Lafferty


Some classes in the Kafka Connect runtime create anonymous inner classes that 
log 'this', resulting in log messages that can't be correlated with any other 
messages. These should scope 'this' to the outer class to have consistent log 
messages.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-367 Introduce close(Duration) to Producer instead of close(long, TimeUnit)

Thanks for the KIP. Makes sense to me. Should we make a similar change to
AdminClient?

-Jason

On Tue, Aug 28, 2018 at 2:32 AM, Chia-Ping Tsai  wrote:

> (re-start the thread for KIP-367 because I enter the incorrect topic in
> first post)
>
> hi all
>
> I would like to start a discussion of KIP-367 [1]. It is similar to
> KIP-358 and KIP-266 which is trying to substitute Duration for (long,
> TimeUnit).
>
> [1] https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=89070496
>
> --
> Chia-Ping
>

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

+1 Thanks for the updates.

On Tue, Aug 28, 2018 at 1:15 AM, Viktor Somogyi-Vass <
viktorsomo...@gmail.com> wrote:

> Sure, I've added it. I'll also do the testing today.
>
> On Mon, Aug 27, 2018 at 5:03 PM Ismael Juma  wrote:
>
> > Thanks Viktor. I think it would be good to verify that existing
> > ExtendedSerializer implementations work without recompiling. This could
> be
> > done as a manual test. If you agree, I suggest adding it to the testing
> > plan section.
> >
> > Ismael
> >
> > On Mon, Aug 27, 2018 at 7:57 AM Viktor Somogyi-Vass <
> > viktorsomo...@gmail.com>
> > wrote:
> >
> > > Thanks guys, I've updated my KIP with this info (so to keep solution
> #1).
> > > If you find it good enough, please vote as well or let me know if you
> > think
> > > something is missing.
> > >
> > > On Sat, Aug 25, 2018 at 1:14 AM Ismael Juma  wrote:
> > >
> > > > I'm OK with 1 too. It makes me a bit sad that we don't have a path
> for
> > > > removing the method without headers, but it seems like the simplest
> and
> > > > least confusing option (I am assuming that headers are not needed in
> > the
> > > > serializers in the common case).
> > > >
> > > > Ismael
> > > >
> > > > On Fri, Aug 24, 2018 at 2:42 PM Jason Gustafson 
> > > > wrote:
> > > >
> > > > > Hey Viktor,
> > > > >
> > > > > Good summary. I agree that option 1) seems like the simplest choice
> > > and,
> > > > as
> > > > > you note, we can always add the default implementation later. I'll
> > > leave
> > > > > Ismael to make a case for the circular forwarding approach ;)
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Fri, Aug 24, 2018 at 3:02 AM, Viktor Somogyi-Vass <
> > > > > viktorsomo...@gmail.com> wrote:
> > > > >
> > > > > > I think in the first draft I didn't provide an implementation for
> > > them
> > > > as
> > > > > > it seemed very simple and straightforward. I looked up a couple
> of
> > > > > > implementations of the ExtendedSerializers on github and the
> > general
> > > > > > behavior seems to be that they delegate to the 2 argument
> > > (headerless)
> > > > > > method:
> > > > > >
> > > > > > https://github.com/khoitnm/practice-spring-kafka-grpc/blob/
> > > > > > a6fc9b3395762c4889807baedd822f4653d5dcdd/kafka-common/src/
> > > > > > main/java/org/tnmk/common/kafka/serialization/protobuf/
> > > > > > ProtobufSerializer.java
> > > > > >
> > > https://github.com/hong-zhu/nxgen/blob/5cf1427d4e1a8f5c7fab47955af32a
> > > > > > 0d4f4873af/nxgen-kafka-client/src/main/java/nxgen/kafka/
> > > > > > client/event/serdes/EventSerializer.java
> > > > > > https://github.com/jerry-jx/spring-kafka/blob/
> > > > > > ac323ec5b8b9a0ca975db2f7322ff6878fce481a/spring-kafka/src/
> > > > > >
> > > > >
> > > >
> > >
> > main/java/org/springframework/kafka/support/serializer/
> JsonSerializer.java
> > > > > > https://github.com/enzobonggio/nonblocking-kafka/blob/
> > > > > > bc1a379b2d9593b46cf9604063bc5b38e2785d19/src/main/java/com/
> > > > > > example/kafka/producer/CustomJsonSerializer.java
> > > > > >
> > > > > > Of course 4 example is not representative but it shows that these
> > > users
> > > > > > usually delegate to the "headerless" (2 argument) method. I've
> > tried
> > > to
> > > > > > look it up on other code search sites but haven't had much luck
> so
> > > far.
> > > > > > Given these examples and the way they implement them I'd say it's
> > > more
> > > > > > common to delegate to the headerless method, that's why I think
> > it's
> > > a
> > > > > good
> > > > > > approach for us too. Now having a default implementation for that
> > is
> > > > > again
> > > > > > a good question. I think current use cases wouldn't change in
> > either
> > > > case
> > > > > > (unless we deprecate the headerless one).
> > > > > > For the new use cases it depends what do we want to propagate
> going
> > > > > > forward. Do we want only one method to exist or two? As Ismael
> > > > > highlighted
> > > > > > it might be confusing if we have 2 methods, both with default
> > > > > > implementation and in this case we want to push the 3 argument
> one
> > > for
> > > > > > users.
> > > > > >
> > > > > > So I see three possible ways:
> > > > > > 1.) Don't provide a default implementation for the headerless
> > method.
> > > > > This
> > > > > > supports the current implementations and encourages the
> delegation
> > > > style
> > > > > in
> > > > > > future implementations. This might be the simplest option.
> > > > > > 2.) Provide a default implementation for the headerless method.
> > This
> > > > > would
> > > > > > be a bit confusing, so we'd likely push the use of the 3
> parameter
> > > > method
> > > > > > and deprecate the headerless. This would however further litter
> the
> > > > code
> > > > > > base with deprecation warnings as we're using the headerless
> method
> > > in
> > > > a
> > > > > > lot of places (think of the current serializers/deserializers).
> So
> > in
> > > > > this
> > > > > > case we would want to clean up the code base a little

Re: [DISCUSS] KIP-359: Verify leader epoch in produce requests

Hey Dong,

Yes, that was the intent. I fixed the proposal. Thanks for reviewing!

-Jason

On Tue, Aug 28, 2018 at 10:23 AM, Dong Lin  wrote:

> Hey Jason,
>
> Thanks for the KIP. The KIP looks good overall.
>
> I have one minor question. The KIP says that "If the producer sees either
> the FENCED_LEADER_EPOCH or the UNKNOWN_LEADER_EPOCH in the response from
> the broker, it will refresh metadata and retry". Given that
> UNKNOWN_LEADER_EPOCH
> is returned to producer iff producer's leaderEpoch > broker's leaderEpoch,
> it probably means the producer's metadata is newer than broker's leadership
> information. In this case, does producer needs to refresh metadata?
>
>
> Thanks,
> Dong
>
>
> On Tue, Aug 28, 2018 at 9:07 AM, Jason Gustafson 
> wrote:
>
> > Hey All,
> >
> > I didn't expect many comments here. This is mostly for consistency and to
> > improve debugging now that we have leader epoch in the metadata. I'll go
> > ahead and start a vote shortly.
> >
> > Thanks,
> > Jason
> >
> > On Sat, Aug 18, 2018 at 2:10 PM, Jason Gustafson 
> > wrote:
> >
> > > Hi All,
> > >
> > > I've added a short KIP to add leader epoch validation to the produce
> API.
> > > This is a follow-up to KIP-320, which added similar protection to the
> > > consumer APIs. Take a look and let me know what you think.
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 359%3A+Verify+leader+epoch+in+produce+requests
> > >
> > > Thanks,
> > > Jason
> > >
> >
>

Build failed in Jenkins: kafka-trunk-jdk10 #440

See 


Changes:

[jason] KAFKA-7128; Follower has to catch up to offset within current leader

--
[...truncated 1.53 MB...]

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAppendNewMetadataToLogOnAddPartitionsWhenPartitionsAdded PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithInvalidRequestOnEndTxnWhenTransactionalIdIsNull STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithInvalidRequestOnEndTxnWhenTransactionalIdIsNull PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithCoordinatorLoadInProgressOnEndTxnWhenCoordinatorIsLoading 
STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithCoordinatorLoadInProgressOnEndTxnWhenCoordinatorIsLoading 
PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithSuccessOnAddPartitionsWhenStateIsCompleteAbort STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithSuccessOnAddPartitionsWhenStateIsCompleteAbort PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldWaitForCommitToCompleteOnHandleInitPidAndExistingTransactionInPrepareAbortState
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldWaitForCommitToCompleteOnHandleInitPidAndExistingTransactionInPrepareAbortState
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithErrorsNoneOnAddPartitionWhenNoErrorsAndPartitionsTheSame 
STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithErrorsNoneOnAddPartitionWhenNoErrorsAndPartitionsTheSame PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithInvalidRequestOnEndTxnWhenTransactionalIdIsEmpty STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithInvalidRequestOnEndTxnWhenTransactionalIdIsEmpty PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithInvalidRequestAddPartitionsToTransactionWhenTransactionalIdIsEmpty
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithInvalidRequestAddPartitionsToTransactionWhenTransactionalIdIsEmpty
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAppendPrepareAbortToLogOnEndTxnWhenStatusIsOngoingAndResultIsAbort STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAppendPrepareAbortToLogOnEndTxnWhenStatusIsOngoingAndResultIsAbort PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAcceptInitPidAndReturnNextPidWhenTransactionalIdIsNull STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAcceptInitPidAndReturnNextPidWhenTransactionalIdIsNull PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRemoveTransactionsForPartitionOnEmigration STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRemoveTransactionsForPartitionOnEmigration PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldWaitForCommitToCompleteOnHandleInitPidAndExistingTransactionInPrepareCommitState
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldWaitForCommitToCompleteOnHandleInitPidAndExistingTransactionInPrepareCommitState
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAbortExpiredTransactionsInOngoingStateAndBumpEpoch STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldAbortExpiredTransactionsInOngoingStateAndBumpEpoch PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldReturnInvalidTxnRequestOnEndTxnRequestWhenStatusIsCompleteCommitAndResultIsNotCommit
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldReturnInvalidTxnRequestOnEndTxnRequestWhenStatusIsCompleteCommitAndResultIsNotCommit
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldReturnOkOnEndTxnWhenStatusIsCompleteCommitAndResultIsCommit STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldReturnOkOnEndTxnWhenStatusIsCompleteCommitAndResultIsCommit PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionsOnAddPartitionsWhenStateIsPrepareCommit 
STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldRespondWithConcurrentTransactionsOnAddPartitionsWhenStateIsPrepareCommit 
PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldIncrementEpochAndUpdateMetadataOnHandleInitPidWhenExistingCompleteTransaction
 STARTED

kafka.coordinator.transaction.TransactionCoordinatorTest > 
shouldIncrementEpochAndUpdateMetadataOnHandleInitPidWhenExistingCompleteTransaction
 PASSED

kafka.coordinator.transaction.TransactionCoordinatorTest >

[jira] [Resolved] (KAFKA-6801) Restrict Consumer to fetch data from secure port only, and deny from non-secure port.



 [ 
https://issues.apache.org/jira/browse/KAFKA-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6801.
--
Resolution: Information Provided

Closing as per above comment. 

> Restrict Consumer to fetch data from secure port only, and deny from 
> non-secure port.
> -
>
> Key: KAFKA-6801
> URL: https://issues.apache.org/jira/browse/KAFKA-6801
> Project: Kafka
>  Issue Type: Task
>  Components: admin, config, consumer, security
>Affects Versions: 0.10.2.1
>Reporter: VinayKumar
>Priority: Major
>
> I have listeners configured with 2 ports as below:  (9092 -> Plaintext, 9092 
> -> SASL_PLAIN)
> listeners=PLAINTEXT://:9092, SASL_PLAIN://:9093
> For a topic, I want restrict Consumers to consume data from 9093 port only, 
> and consuming data from 9092 port should be denied.
> I've gone through ACL concept, but haven't seen an option to restrict 
> Consumer pulling data from non-secure port (in this case- 9092)
> Can someone please let me know if this is configurable ?
> Can my requirement be fulfilled. Please provide necessary info.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-7242) Externalized secrets are revealed in task configuration

2018-08-28 Thread Ewen Cheslack-Postava (JIRA)



 [ 
https://issues.apache.org/jira/browse/KAFKA-7242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-7242.
--
   Resolution: Fixed
Fix Version/s: 2.1.0
   2.0.1

Issue resolved by pull request 5475
[https://github.com/apache/kafka/pull/5475]

> Externalized secrets are revealed in task configuration
> ---
>
> Key: KAFKA-7242
> URL: https://issues.apache.org/jira/browse/KAFKA-7242
> Project: Kafka
>  Issue Type: Bug
>Reporter: Bahdan Siamionau
>Assignee: Robert Yokota
>Priority: Major
> Fix For: 2.0.1, 2.1.0
>
>
> Trying to use new [externalized 
> secrets|https://issues.apache.org/jira/browse/KAFKA-6886] feature I noticed 
> that task configuration is being saved in config topic with disclosed 
> secrets. It seems like the main goal of feature was not achieved - secrets 
> are still persisted in plain-text. Probably I'm misusing this new config, 
> please correct me if I wrong.
> I'm running connect in distributed mode, creating connector with following 
> config:
> {code:java}
> {
>   "name" : "jdbc-sink-test",
>   "config" : {
> "connector.class" : "io.confluent.connect.jdbc.JdbcSinkConnector",
> "tasks.max" : "1",
> "config.providers" : "file",
> "config.providers.file.class" : 
> "org.apache.kafka.common.config.provider.FileConfigProvider",
> "config.providers.file.param.secrets" : "/opt/mysecrets",
> "topics" : "test_topic",
> "connection.url" : "${file:/opt/mysecrets:url}",
> "connection.user" : "${file:/opt/mysecrets:user}",
> "connection.password" : "${file:/opt/mysecrets:password}",
> "insert.mode" : "upsert",
> "pk.mode" : "record_value",
> "pk.field" : "id"
>   }
> }
> {code}
> Connector works fine, placeholders are substituted with correct values from 
> file, but then updated config is written into  the topic again (see 3 
> following records in config topic):
> {code:java}
> key: connector-jdbc-sink-test
> value:
> {
> "properties": {
> "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
> "tasks.max": "1",
> "config.providers": "file",
> "config.providers.file.class": 
> "org.apache.kafka.common.config.provider.FileConfigProvider",
> "config.providers.file.param.secrets": "/opt/mysecrets",
> "topics": "test_topic",
> "connection.url": "${file:/opt/mysecrets:url}",
> "connection.user": "${file:/opt/mysecrets:user}",
> "connection.password": "${file:/opt/mysecrets:password}",
> "insert.mode": "upsert",
> "pk.mode": "record_value",
> "pk.field": "id",
> "name": "jdbc-sink-test"
> }
> }
> key: task-jdbc-sink-test-0
> value:
> {
> "properties": {
> "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
> "config.providers.file.param.secrets": "/opt/mysecrets",
> "connection.password": "actualpassword",
> "tasks.max": "1",
> "topics": "test_topic",
> "config.providers": "file",
> "pk.field": "id",
> "task.class": "io.confluent.connect.jdbc.sink.JdbcSinkTask",
> "connection.user": "datawarehouse",
> "name": "jdbc-sink-test",
> "config.providers.file.class": 
> "org.apache.kafka.common.config.provider.FileConfigProvider",
> "connection.url": 
> "jdbc:postgresql://actualurl:5432/datawarehouse?stringtype=unspecified",
> "insert.mode": "upsert",
> "pk.mode": "record_value"
> }
> }
> key: commit-jdbc-sink-test
> value:
> {
> "tasks":1
> }
> {code}
> Please advice have I misunderstood the goal of the given feature, have I 
> missed smth in configuration or is it actually a bug? Thank you



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-5962) java.io.IOException: Map failed



 [ 
https://issues.apache.org/jira/browse/KAFKA-5962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-5962.
--
Resolution: Fixed

Closing as docs added in KAFKA-6343

> java.io.IOException: Map failed
> ---
>
> Key: KAFKA-5962
> URL: https://issues.apache.org/jira/browse/KAFKA-5962
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.11.0.0
> Environment: kafka_2.12-0.11.0.0
>Reporter: Mehmet Soner
>Priority: Critical
> Attachments: broker-log-failed-during-restart.txt, 
> broker-log-start-and-shutdown.txt
>
>
> *OS:* HP-UX B.11.31 U ia64
> Step to reproduce bug:
> *1) Starting zookeeper by using below command.*
> zookeeper-server-start.sh -daemon 
> /usr/local/Apache/kafka_2.12-0.11.0.0/config/zookeeper.properties
> *2) Starting kafka by using below command.*
> kafka-server-start.sh -daemon 
> /usr/local/Apache/kafka_2.12-0.11.0.0/config/server0.properties
> *3) Writing data to topic*
> /prov/users/sas/bin$ $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list 
> 172.31.19.85:9092 --topic ssmtopup_topic
> >test1
> >test2
> >test3
> >test4
> >test5
> *server.log*
> [2017-09-22 09:51:24,467] INFO Updated PartitionLeaderEpoch. New: {epoch:7, 
> offset:0}, Current: {epoch:-1, offset-1} for Partition: ssmtopup_topic-0. 
> Cache now contains 0 entries. (kafka.server.epoch.LeaderEpochFileCache)
> [2017-09-22 09:52:23,819] INFO [Group Metadata Manager on Broker 0]: Removed 
> 0 expired offsets in 0 milliseconds. 
> (kafka.coordinator.group.GroupMetadataManager)
> *4) Exiting by pressing CTRL+C*
> *5) Shutting down kafka by using below command*
> *a) finding pid by uisin below command*
> jps -lm
> *b) graceful shutdown*
> kill -15 
> *You can find error below*
> *server.log *
> [2017-09-22 09:52:26,179] INFO [Kafka Server 0], shutting down 
> (kafka.server.KafkaServer)
> [2017-09-22 09:52:26,219] INFO [Kafka Server 0], Starting controlled shutdown 
> (kafka.server.KafkaServer)
> [2017-09-22 09:52:26,346] INFO [Kafka Server 0], Controlled shutdown 
> succeeded (kafka.server.KafkaServer)
> [2017-09-22 09:52:26,356] INFO [Socket Server on Broker 0], Shutting down 
> (kafka.network.SocketServer)
> [2017-09-22 09:52:26,384] INFO [Socket Server on Broker 0], Shutdown 
> completed (kafka.network.SocketServer)
> [2017-09-22 09:52:26,386] INFO [Kafka Request Handler on Broker 0], shutting 
> down (kafka.server.KafkaRequestHandlerPool)
> [2017-09-22 09:52:26,395] INFO [Kafka Request Handler on Broker 0], shut down 
> completely (kafka.server.KafkaRequestHandlerPool)
> [2017-09-22 09:52:26,411] INFO [ThrottledRequestReaper-Fetch]: Shutting down 
> (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
> [2017-09-22 09:52:27,032] INFO [ThrottledRequestReaper-Fetch]: Stopped 
> (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
> [2017-09-22 09:52:27,034] INFO [ThrottledRequestReaper-Fetch]: Shutdown 
> completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
> [2017-09-22 09:52:27,034] INFO [ThrottledRequestReaper-Produce]: Shutting 
> down (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
> [2017-09-22 09:52:27,701] INFO [ThrottledRequestReaper-Produce]: Shutdown 
> completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
> [2017-09-22 09:52:27,701] INFO [ThrottledRequestReaper-Produce]: Stopped 
> (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
> [2017-09-22 09:52:27,701] INFO [ThrottledRequestReaper-Request]: Shutting 
> down (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
> [2017-09-22 09:52:27,985] INFO [ThrottledRequestReaper-Request]: Shutdown 
> completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
> [2017-09-22 09:52:27,985] INFO [ThrottledRequestReaper-Request]: Stopped 
> (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
> [2017-09-22 09:52:27,988] INFO [KafkaApi-0] Shutdown complete. 
> (kafka.server.KafkaApis)
> [2017-09-22 09:52:27,991] INFO [ExpirationReaper-0-topic]: Shutting down 
> (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
> [2017-09-22 09:52:28,082] INFO [ExpirationReaper-0-topic]: Stopped 
> (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
> [2017-09-22 09:52:28,088] INFO [ExpirationReaper-0-topic]: Shutdown completed 
> (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
> [2017-09-22 09:52:28,096] INFO [Transaction Coordinator 0]: Shutting down. 
> (kafka.coordinator.transaction.TransactionCoordinator)
> [2017-09-22 09:52:28,100] INFO [ProducerId Manager 0]: Shutdown complete: 
> last producerId assigned 7000 
> (kafka.coordinator.transaction.ProducerIdManager)
> [2017-09-22 09:52:28,102] INFO [Transaction State Manager 0]: Shutdown 
> complete (kafka.coordinator.transaction.TransactionStateManager)
> [2017-09-22 09:52:28,102] INFO [Transaction

[jira] [Resolved] (KAFKA-1712) Excessive storage usage on newly added node



 [ 
https://issues.apache.org/jira/browse/KAFKA-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-1712.
--
Resolution: Fixed

Fixed via https://issues.apache.org/jira/browse/KAFKA-2511

> Excessive storage usage on newly added node
> ---
>
> Key: KAFKA-1712
> URL: https://issues.apache.org/jira/browse/KAFKA-1712
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Reporter: Oleg Golovin
>Priority: Major
>
> When a new node is added to cluster data starts replicating into it. The 
> mtime of creating segments will be set on the last message being written to 
> them. Though the replication is a prolonged process, let's assume (for 
> simplicity of explanation) that their mtime is very close to the time when 
> the new node was added.
> After the replication is done, new data will start to flow into this new 
> node. After `log.retention.hours` the amount of data will be 2 * 
> daily_amount_of_data_in_kafka_node (first one is the replicated data from 
> other nodes when the node was added (let us call it `t1`) and the second is 
> the amount of replicated data from other nodes which happened from `t1` to 
> `t1 + log.retention.hours`). So by that time the node will have twice as much 
> data as the other nodes.
> This poses a big problem to us as our storage is chosen to fit normal amount 
> of data (not twice this amount).
> In our particular case it poses another problem. We have an emergency segment 
> cleaner which runs in case storage is nearly full (>90%). We try to balance 
> the amount of data for it not to run to rely solely on kafka internal log 
> deletion, but sometimes emergency cleaner runs.
> It works this way:
> - it gets all kafka segments for the volume
> - it filters out last segments of each partition (just to avoid unnecessary 
> recreation of last small-size segments)
> - it sorts them by segment mtime
> - it changes mtime of the first N segements (with the lowest mtime) to 1, so 
> they become really really old. Number N is chosen to free specified 
> percentage of volume (3% in our case).  Kafka deletes these segments later 
> (as they are very old).
> Emergency cleaner works very well. Except for the case when the data is 
> replicated to the newly added node. 
> In this case segment mtime is the time the segment was replicated and does 
> not reflect the real creation time of original data stored in this segment.
> So in this case kafka emergency cleaner will delete segments with the lowest 
> mtime, which may hold the data which is much more recent than the data in 
> other segments.
> This is not a big problem until we delete the data which hasn't been fully 
> consumed.
> In this case we loose data and this makes it a big problem.
> Is it possible to retain segment mtime during initial replication on a new 
> node?
> This will help not to load the new node with the twice as large amount of 
> data as other nodes have.
> Or maybe there are another ways to sort segments by data creation times (or 
> close to data creation time)? (for example if this ticket is implemented 
> https://issues.apache.org/jira/browse/KAFKA-1403, we may take time of the 
> first message from .index). In our case it will help with kafka emergency 
> cleaner, which will be deleting really the oldest data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-1665) controller state gets stuck in message after execute



 [ 
https://issues.apache.org/jira/browse/KAFKA-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-1665.
--
Resolution: Auto Closed

Closing inactive issue. Closing as per above comments.

> controller state gets stuck in message after execute
> 
>
> Key: KAFKA-1665
> URL: https://issues.apache.org/jira/browse/KAFKA-1665
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Reporter: Joe Stein
>Priority: Major
>
> I had a 0.8.1.1 Kafka Broker go down, and I was trying to use the reassign 
> partition script to move topics off that broker. When I describe the topics, 
> I see the following:
> Topic: mini__022active_120__33__mini Partition: 0 Leader: 2131118 
> Replicas: 2131118,2166601,2163421 Isr: 2131118,2166601
> This shows that the broker “2163421” is down. So I create the following file 
> /tmp/move_topic.json:
> {
> "version": 1,
> "partitions": [
> {
> "topic": "mini__022active_120__33__mini",
> "partition": 0,
> "replicas": [
> 2131118, 2166601,  2156998
> ]
> }
> ]
> }
> And then do this:
> ./kafka-reassign-partitions.sh --execute --reassignment-json-file 
> /tmp/move_topic.json
> Successfully started reassignment of partitions 
> {"version":1,"partitions":[{"topic":"mini__022active_120__33__mini","partition":0,"replicas":[2131118,2166601,2156998]}]}
> However, when I try to verify this, I get the following error:
> ./kafka-reassign-partitions.sh --verify --reassignment-json-file 
> /tmp/move_topic.json
> Status of partition reassignment:
> ERROR: Assigned replicas (2131118,2166601,2156998,2163421) don't match the 
> list of replicas for reassignment (2131118,2166601,2156998) for partition 
> [mini__022active_120__33__mini,0]
> Reassignment of partition [mini__022active_120__33__mini,0] failed
> If I describe the topics, I now see there are 4 replicas. This has been like 
> this for many hours now, so it seems to have permanently moved to 4 replicas 
> for some reason.
> Topic:mini__022active_120__33__mini PartitionCount:1 ReplicationFactor:4 
> Configs:
> Topic: mini__022active_120__33__mini Partition: 0 Leader: 2131118 
> Replicas: 2131118,2166601,2156998,2163421 Isr: 2131118,2166601
> If I re-execute and re-verify, I get the same error. So it seems to be wedged.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-2127) Running TopicCommand --alter causes connection close/reset errors in kafka logs



 [ 
https://issues.apache.org/jira/browse/KAFKA-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2127.
--
Resolution: Auto Closed

Closing inactive issue. Please reopen if the issue still exists in newer 
versions.

> Running TopicCommand --alter causes connection close/reset errors in kafka 
> logs
> ---
>
> Key: KAFKA-2127
> URL: https://issues.apache.org/jira/browse/KAFKA-2127
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Jason Rosenberg
>Priority: Minor
>
> I am using 0.8.2.1.  I've been noticing that any time I use TopicCommand to 
> alter a topic (e.g. add partitions) or delete a topic, the broker logs show a 
> bunch of closed connections, and usually 2 or 3 Connection reset exceptions.  
> It logs these with ERROR status.
> I recently used the kafka.admin.TopicCommand tool to increase the partitions 
> for a topic from 1 to 4.  So I ran:
> {code}
>  java -cp kafka.jar kafka.admin.TopicCommand --zookeeper myzkserver:12345 
> --topic mytopic --alter --partitions 4
> {code}
> This resulted in the following sequence in the broker log (repeated pretty 
> much in the logs of each broker):
> {code}
> 2015-04-16 03:51:26,156  INFO [kafka-network-thread-27330-1] 
> network.Processor - Closing socket connection to /1.2.3.12.
> 2015-04-16 03:51:26,169  INFO [kafka-network-thread-27330-0] 
> network.Processor - Closing socket connection to /1.2.3.89.
> 2015-04-16 03:51:26,169  INFO [kafka-network-thread-27330-0] 
> network.Processor - Closing socket connection to /1.2.3.95.
> 2015-04-16 03:51:26,176 ERROR [kafka-network-thread-27330-2] 
> network.Processor - Closing socket for /1.2.4.34 because of error
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
> at kafka.utils.Utils$.read(Utils.scala:380)
> at 
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
> at kafka.network.Processor.read(SocketServer.scala:444)
> at kafka.network.Processor.run(SocketServer.scala:340)
> at java.lang.Thread.run(Thread.java:745)
> 2015-04-16 03:51:26,178 ERROR [kafka-network-thread-27330-1] 
> network.Processor - Closing socket for /1.2.4.59 because of error
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
> at kafka.utils.Utils$.read(Utils.scala:380)
> at 
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
> at kafka.network.Processor.read(SocketServer.scala:444)
> at kafka.network.Processor.run(SocketServer.scala:340)
> at java.lang.Thread.run(Thread.java:745)
> 2015-04-16 03:51:26,192 ERROR [kafka-network-thread-27330-1] 
> network.Processor - Closing socket for /1.2.3.11 because of error
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
> at kafka.utils.Utils$.read(Utils.scala:380)
> at 
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
> at kafka.network.Processor.read(SocketServer.scala:444)
> at kafka.network.Processor.run(SocketServer.scala:340)
> at java.lang.Thread.run(Thread.java:745)
> 2015-04-16 03:51:26,451  INFO [kafka-request-handler-3] 
> server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 45] Removed 
> fetcher for partitions [mytopic,2]
> 2015-04-16 03:51:26,453  INFO [kafka-request-handler-3] log.Log - Completed 
> load of log mytopic-2 with log end offset 0
> 2015-04-16 03:51:26,454  INFO [kafka-request-handler-3] log.LogManager - 
> Created log for partition [mytopic,2] in /data/kafka_logs with properties 
> {segment.index.bytes -> 10485760, file.delete.delay.ms -> 6, 
> segment.bytes -> 1073741824, flush.ms -> 9223372036854775807, 
> delete.retention.ms -> 8640, index.interval.bytes -> 4096, 
> retention.bytes -> 500,

[jira] [Resolved] (KAFKA-7291) kafka-console-consumer.sh waits on inexistent topic



 [ 
https://issues.apache.org/jira/browse/KAFKA-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-7291.
--
Resolution: Fixed

Fixed in newer versions.  Yes, you need to upgrade to latest versions.

> kafka-console-consumer.sh waits on inexistent topic
> ---
>
> Key: KAFKA-7291
> URL: https://issues.apache.org/jira/browse/KAFKA-7291
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.8.2.2
>Reporter: HUSLEAG Dumitru
>Priority: Major
>
> Hello,
> My request concerns kafka-console-consumer.sh behavior (which I suppose is 
> based on KafkaConsumer).
> If I try to consume from a topic that does not exist it connects and waits as 
> if it existed, which I find is illogical.
> The broker does not have enabled *{{auto.create.topics.enable}}*
> {{Let's say I launch this command:}}
> {code}
> ./kafka-console-consumer.sh --zookeeper $(hostname):2181/kafka --topic 
> inexistentTopic
> {code}
> and inexistentTopic does not exist indeed, then the kafka-console-consumer.sh 
> will wait forever instead of exit with error code and display an error 
> message saying that the topic does not exist. 
> Anyway that's the way I would expect it to behave in this case.
> Please consider this request.
> Regards,
> Dumitru
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-7128) Lagging high watermark can lead to committed data loss after ISR expansion

2018-08-28 Thread Jason Gustafson (JIRA)



 [ 
https://issues.apache.org/jira/browse/KAFKA-7128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-7128.

   Resolution: Fixed
Fix Version/s: 2.1.0
   2.0.1
   1.1.2

> Lagging high watermark can lead to committed data loss after ISR expansion
> --
>
> Key: KAFKA-7128
> URL: https://issues.apache.org/jira/browse/KAFKA-7128
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Anna Povzner
>Priority: Major
> Fix For: 1.1.2, 2.0.1, 2.1.0
>
>
> Some model checking exposed a weakness in the ISR expansion logic. We know 
> that the high watermark can go backwards after a leader failover, but we may 
> not have known that this can lead to the loss of committed data.
> Say we have three replicas: r1, r2, and r3. Initially, the ISR consists of 
> (r1, r2) and the leader is r1. r3 is a new replica which has not begun 
> fetching. The data up to offset 10 has been committed to the ISR. Here is the 
> initial state:
> State 1
> ISR: (r1, r2)
>  Leader: r1
>  r1: [hw=10, leo=10]
>  r2: [hw=5, leo=10]
>  r3: [hw=0, leo=0]
> Replica 1 then initiates shutdown (or fails) and leaves the ISR, which makes 
> r2 the new leader. The high watermark is still lagging r1.
> State 2
> ISR: (r2)
>  Leader: r2
>  r1 (offline): [hw=10, leo=10]
>  r2: [hw=5, leo=10]
>  r3: [hw=0, leo=0]
> Replica 3 then catch up to the high watermark on r2 and joins the ISR. 
> Perhaps it's high watermark is lagging behind r2, but this is unimportant.
> State 3
> ISR: (r2, r3)
>  Leader: r2
>  r1 (offline): [hw=10, leo=10]
>  r2: [hw=5, leo=10]
>  r3: [hw=0, leo=5]
> Now r2 fails and r3 is elected leader and is the only member of the ISR. The 
> committed data from offsets 5 to 10 has been lost.
> State 4
> ISR: (r3)
>  Leader: r3
>  r1 (offline): [hw=10, leo=10]
>  r2 (offline): [hw=5, leo=10]
>  r3: [hw=0, leo=5]
> The bug is the fact that we allowed r3 into the ISR after the local high 
> watermark had been reached. Since the follower does not know the true high 
> watermark for the previous leader's epoch, it should not allow a replica to 
> join the ISR until it has caught up to an offset within its own epoch.
> Note this is related to 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-207%3A+Offsets+returned+by+ListOffsetsResponse+should+be+monotonically+increasing+even+during+a+partition+leader+change]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-359: Verify leader epoch in produce requests

2018-08-28 Thread Dong Lin

Hey Jason,

Thanks for the KIP. The KIP looks good overall.

I have one minor question. The KIP says that "If the producer sees either
the FENCED_LEADER_EPOCH or the UNKNOWN_LEADER_EPOCH in the response from
the broker, it will refresh metadata and retry". Given that
UNKNOWN_LEADER_EPOCH
is returned to producer iff producer's leaderEpoch > broker's leaderEpoch,
it probably means the producer's metadata is newer than broker's leadership
information. In this case, does producer needs to refresh metadata?

Thanks,
Dong

On Tue, Aug 28, 2018 at 9:07 AM, Jason Gustafson  wrote:

> Hey All,
>
> I didn't expect many comments here. This is mostly for consistency and to
> improve debugging now that we have leader epoch in the metadata. I'll go
> ahead and start a vote shortly.
>
> Thanks,
> Jason
>
> On Sat, Aug 18, 2018 at 2:10 PM, Jason Gustafson 
> wrote:
>
> > Hi All,
> >
> > I've added a short KIP to add leader epoch validation to the produce API.
> > This is a follow-up to KIP-320, which added similar protection to the
> > consumer APIs. Take a look and let me know what you think.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 359%3A+Verify+leader+epoch+in+produce+requests
> >
> > Thanks,
> > Jason
> >
>

Re: [DISCUSS] KIP-359: Verify leader epoch in produce requests

Hey All,

I didn't expect many comments here. This is mostly for consistency and to
improve debugging now that we have leader epoch in the metadata. I'll go
ahead and start a vote shortly.

Thanks,
Jason

On Sat, Aug 18, 2018 at 2:10 PM, Jason Gustafson  wrote:

> Hi All,
>
> I've added a short KIP to add leader epoch validation to the produce API.
> This is a follow-up to KIP-320, which added similar protection to the
> consumer APIs. Take a look and let me know what you think.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 359%3A+Verify+leader+epoch+in+produce+requests
>
> Thanks,
> Jason
>

[DISCUSS] KIP 368: Allow SASL Connections to Periodically Re-Authenticate

2018-08-28 Thread Ron Dagostino

Hi everyone. I created KIP 368: Allow SASL Connections to Periodically
Re-Authenticate

(
https://cwiki.apache.org/confluence/display/KAFKA/KIP-368%3A+Allow+SASL+Connections+to+Periodically+Re-Authenticate).
The motivation for this KIP is as follows:

The adoption of KIP-255: OAuth Authentication via SASL/OAUTHBEARER
in
release 2.0.0 creates the possibility of using information in the bearer
token to make authorization decisions. Unfortunately, however, Kafka
connections are long-lived, so there is no ability to change the bearer
token associated with a particular connection. Allowing SASL connections
to periodically re-authenticate would resolve this. In addition to this
motivation there are two others that are security-related. First, to
eliminate access to Kafka for connected clients, the current requirement is
to remove all authorizations (i.e. remove all ACLs). This is necessary
because of the long-lived nature of the connections. It is operationally
simpler to shut off access at the point of authentication, and with the
release of KIP-86: Configurable SASL Callback Handlers

it
is going to become more and more likely that installations will
authenticate users against external directories (e.g. via LDAP). The
ability to stop Kafka access by simply disabling an account in an LDAP
directory (for example) is desirable. The second motivating factor for
re-authentication related to security is that the use of short-lived tokens
is a common OAuth security recommendation, but issuing a short-lived token
to a Kafka client (or a broker when OAUTHBEARER is the inter-broker
protocol) currently has no benefit because once a client is connected to a
broker the client is never challenged again and the connection may remain
intact beyond the token expiration time (and may remain intact indefinitely
under perfect circumstances). This KIP proposes adding the ability for
clients (and brokers when OAUTHBEARER is the inter-broker protocol) to
re-authenticate their connections to brokers and have the new bearer token
appear on their session rather than the old one.

The description of this KIP is actually quite straightforward from a
functionality perspective; from an implementation perspective, though, the
KIP is not so straightforward, so it includes a pull request with a
proposed implementation. See https://github.com/apache/kafka/pull/5582.

Ron

[jira] [Created] (KAFKA-7352) KIP-368: Allow SASL Connections to Periodically Re-Authenticate

2018-08-28 Thread Ron Dagostino (JIRA)

Ron Dagostino created KAFKA-7352:


 Summary: KIP-368: Allow SASL Connections to Periodically 
Re-Authenticate
 Key: KAFKA-7352
 URL: https://issues.apache.org/jira/browse/KAFKA-7352
 Project: Kafka
  Issue Type: Improvement
  Components: clients, core
Reporter: Ron Dagostino
Assignee: Ron Dagostino


KIP-368: Allow SASL Connections to Periodically Re-Authenticate

The adoption of KIP-255: OAuth Authentication via SASL/OAUTHBEARER in release 
2.0.0 creates the possibility of using information in the bearer token to make 
authorization decisions.  Unfortunately, however, Kafka connections are 
long-lived, so there is no ability to change the bearer token associated with a 
particular connection.  Allowing SASL connections to periodically 
re-authenticate would resolve this.  In addition to this motivation there are 
two others that are security-related.  First, to eliminate access to Kafka the 
current requirement is to remove all authorizations (i.e. remove all ACLs).  
This is necessary because of the long-lived nature of the connections.  It is 
operationally simpler to shut off access at the point of authentication, and 
with the release of KIP-86: Configurable SASL Callback Handlers it is going to 
become more and more likely that installations will authenticate users against 
external directories (e.g. via LDAP).  The ability to stop Kafka access by 
simply disabling an account in an LDAP directory (for example) is desirable.  
The second motivating factor for re-authentication related to security is that 
the use of short-lived tokens is a common OAuth security recommendation, but 
issuing a short-lived token to a Kafka client (or a broker when OAUTHBEARER is 
the inter-broker protocol) currently has no benefit because once a client is 
connected to a broker the client is never challenged again and the connection 
may remain intact beyond the token expiration time (and may remain intact 
indefinitely under perfect circumstances).  This KIP proposes adding the 
ability for clients (and brokers when OAUTHBEARER is the inter-broker protocol) 
to re-authenticate their connections to brokers and have the new bearer token 
appear on their session rather than the old one.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: KIP-213 - Scalable/Usable Foreign-Key KTable joins - Rebooted.

2018-08-28 Thread Adam Bellemare

Okay, I will implement John's suggestion of namespacing the external
headers prior to processing, and then removing the namespacing prior to
emitting. A potential future KIP could be to provide this namespacing
automatically.

I would also appreciate any other additional comments on the workflow. My
goal is suss out agreement prior to moving to a vote.

On Mon, Aug 27, 2018 at 3:19 PM, Guozhang Wang  wrote:

> I like John's idea as well: for this KIP specifically as we do not expect
> any other consumers to read the repartition topics externally, we can
> slightly prefix the header to be safe, while keeping the additional cost
> (note the header field is per-record, so any additional byte is per-record
> as well) low.
>
>
> Guozhang
>
> On Tue, Aug 21, 2018 at 11:58 AM, Adam Bellemare  >
> wrote:
>
> > Hi John
> >
> > That is an excellent idea. The header usage I propose would be limited
> > entirely to internal topics, and this could very well be the solution to
> > potential conflicts. If we do not officially reserve a prefix "__" then I
> > think this would be the safest idea, as it would entirely avoid any
> > accidents (perhaps if a company is using its own "__" prefix for other
> > reasons).
> >
> > Thanks
> >
> > Adam
> >
> >
> > On Tue, Aug 21, 2018 at 2:24 PM, John Roesler  wrote:
> >
> > > Just a quick thought regarding headers:
> > > > I think there is no absolute-safe ways to avoid conflicts, but we can
> > > still
> > > > consider using some name patterns to reduce the likelihood as much as
> > > > possible.. e.g. consider sth. like the internal topics naming: e.g.
> > > > "__internal_[name]"?
> > >
> > > I think there is a safe way to avoid conflicts, since these headers are
> > > only needed in internal topics (I think):
> > > For internal and changelog topics, we can namespace all headers:
> > > * user-defined headers are namespaced as "external." + headerKey
> > > * internal headers are namespaced as "internal." + headerKey
> > >
> > > This is a lot of characters, so we could use a sigil instead (e.g., "_"
> > for
> > > internal, "~" for external)
> > >
> > > We simply apply the namespacing when we read user headers from external
> > > topics into the topology and then de-namespace them before we emit them
> > to
> > > an external topic (via "to" or "through").
> > > Now, it is not possible to collide with user-defined headers.
> > >
> > > That said, I'd also be fine with just reserving "__" as a header prefix
> > and
> > > not worrying about collisions.
> > >
> > > Thanks for the KIP,
> > > -John
> > >
> > > On Tue, Aug 21, 2018 at 9:48 AM Jan Filipiak  >
> > > wrote:
> > >
> > > > Still havent completly grabbed it.
> > > > sorry will read more
> > > >
> > > > On 17.08.2018 21:23, Jan Filipiak wrote:
> > > > > Cool stuff.
> > > > >
> > > > > I made some random remarks. Did not touch the core of the algorithm
> > > yet.
> > > > >
> > > > > Will do Monday 100%
> > > > >
> > > > > I don't see Interactive Queries :) like that!
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 17.08.2018 20:28, Adam Bellemare wrote:
> > > > >> I have submitted a PR with my code against trunk:
> > > > >> https://github.com/apache/kafka/pull/5527
> > > > >>
> > > > >> Do I continue on this thread or do we begin a new one for
> > discussion?
> > > > >>
> > > > >> On Thu, Aug 16, 2018 at 1:40 AM, Jan Filipiak <
> > > jan.filip...@trivago.com
> > > > >
> > > > >> wrote:
> > > > >>
> > > > >>> even before message headers, the option for me always existed to
> > > > >>> just wrap
> > > > >>> the messages into my own custom envelop.
> > > > >>> So I of course thought this through. One sentence in your last
> > email
> > > > >>> triggered all the thought process I put in the back then
> > > > >>> again to design it in the, what i think is the "kafka-way". It
> > ended
> > > up
> > > > >>> ranting a little about what happened in the past.
> > > > >>>
> > > > >>> I see plenty of colleagues of mine falling into traps in the API,
> > > > >>> that I
> > > > >>> did warn about in the 1.0 DSL rewrite. I have the same
> > > > >>> feeling again. So I hope it gives you some insights into my
> though
> > > > >>> process. I am aware that since i never ported 213 to higher
> > > > >>> streams version, I don't really have a steak here and initially I
> > > > >>> didn't
> > > > >>> feel like actually sending it. But maybe you can pull
> > > > >>> something good from it.
> > > > >>>
> > > > >>>   Best jan
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> On 15.08.2018 04:44, Adam Bellemare wrote:
> > > > >>>
> > > >  @Jan
> > > >  Thanks Jan. I take it you mean "key-widening" somehow includes
> > > >  information
> > > >  about which record is processed first? I understand about a
> > > >  CombinedKey
> > > >  with both the Foreign and Primary key, but I don't see how you
> > track
> > > >  ordering metadata in there unless you actually included a
> metadata
> > > >  field
> > > >  in
> > > >

[jira] [Created] (KAFKA-7351) Wrong logic in method ClusterConnectionStates.isDisconnected

2018-08-28 Thread Boris Zhguchev (JIRA)

Boris Zhguchev created KAFKA-7351:
-

 Summary: Wrong logic in method 
ClusterConnectionStates.isDisconnected
 Key: KAFKA-7351
 URL: https://issues.apache.org/jira/browse/KAFKA-7351
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 2.0.0
Reporter: Boris Zhguchev


I think the method ClusterConnectionStates.isDisconnected has wrong logic in 
proccesing wrong ids.
{code:java}
@Before
public void setup() {
this.connectionStates = new ClusterConnectionStates(10_000, 60_000);
}

@Test
public void testIsDisconnected(){
boolean connected = connectionStates.isConnected("fake_node");
boolean disconnected = connectionStates.isDisconnected("fake_node");

assertFalse(connected); // false
assertFalse(disconnected); // false
}
{code}
It can be related with that that code block:
{code:java}
public boolean isDisconnected(String id) {
  NodeConnectionState state = nodeState.get(id);
// may be better is state == null ? true : state.isDisconnected()
  return state != null && state.state.isDisconnected(); 
}
{code}
[link|https://github.com/apache/kafka/blob/19b8ac55c389e4b2022476431a28c8431caed52a/clients/src/main/java/org/apache/kafka/clients/ClusterConnectionStates.java#L252]
 to github



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[DISCUSS] KIP-367 Introduce close(Duration) to Producer instead of close(long, TimeUnit)

2018-08-28 Thread Chia-Ping Tsai

(re-start the thread for KIP-367 because I enter the incorrect topic in first 
post)

hi all

I would like to start a discussion of KIP-367 [1]. It is similar to KIP-358 and 
KIP-266 which is trying to substitute Duration for (long, TimeUnit).

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89070496

--
Chia-Ping

RE: [VOTE] KIP-235 Add DNS alias support for secured connection

2018-08-28 Thread Skrzypek, Jonathan

Hi,

Could someone take a look at https://github.com/apache/kafka/pull/4485 and 
merge if ok ?

Jonathan Skrzypek

-Original Message-
From: Skrzypek, Jonathan [Tech]
Sent: 27 June 2018 17:52
To: dev
Subject: RE: [VOTE] KIP-235 Add DNS alias support for secured connection

Hi,

I've modified the PR last week following comments on unit tests, could it be 
reviewed ?

https://github.com/apache/kafka/pull/4485

Jonathan Skrzypek

-Original Message-
From: Ismael Juma [mailto:ism...@juma.me.uk]
Sent: 23 May 2018 01:29
To: dev
Subject: Re: [VOTE] KIP-235 Add DNS alias support for secured connection

Thanks for the KIP. I think this is a good and low risk change. It would be
good to ensure that it works well with KIP-302 if we think that makes sense
too. In any case, +1 (binding).

Ismael

On Fri, Mar 23, 2018 at 12:05 PM Skrzypek, Jonathan <
jonathan.skrzy...@gs.com> wrote:

> Hi,
>
> I would like to start a vote for KIP-235
>
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D235-253A-2BAdd-2BDNS-2Balias-2Bsupport-2Bfor-2Bsecured-2Bconnection=DwIBaQ=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4=nNmJlu1rR_QFAPdxGlafmDu9_r6eaCbPOM0NM1EHo-E=uPuVydDxaxC8XfuCt8ZC6C93Gx50DlpAJaTqvC80Z_0=KJTm2ESwlBAOOKVyS_Cbt_9WdGyazwlxdWFCvkEvtd4=
>
> This is a proposition to add an option for reverse dns lookup of
> bootstrap.servers hosts, allowing the use of dns aliases on clusters using
> SASL authentication.
>
>
>
>

Your Personal Data: We may collect and process information about you that may 
be subject to data protection laws. For more information about how we use and 
disclose your personal data, how we protect your information, our legal basis 
to use your information, your rights and who you can contact, please refer to: 
www.gs.com/privacy-notices

Re: [DISCUSS] KIP-367 Introduce close(Duration) to Producer instead of close(long, TimeUnit)

2018-08-28 Thread Chia-Ping Tsai

fix the topic of this discussion :(

On 2018/08/28 08:59:36, Chia-Ping Tsai  wrote: 
> hi all
> 
> I would like to start a discussion of KIP-367 [1]. It is similar to KIP-358 
> and KIP-266 which is trying to substitute Duration for (long, TimeUnit).
> 
> [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89070496
> 
> --
> Chia-Ping
>

[DISCUSS]

2018-08-28 Thread Chia-Ping Tsai

hi all

I would like to start a discussion of KIP-367 [1]. It is similar to KIP-358 and 
KIP-266 which is trying to substitute Duration for (long, TimeUnit).

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89070496

--
Chia-Ping

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

2018-08-28 Thread Viktor Somogyi-Vass

Sure, I've added it. I'll also do the testing today.

On Mon, Aug 27, 2018 at 5:03 PM Ismael Juma  wrote:

> Thanks Viktor. I think it would be good to verify that existing
> ExtendedSerializer implementations work without recompiling. This could be
> done as a manual test. If you agree, I suggest adding it to the testing
> plan section.
>
> Ismael
>
> On Mon, Aug 27, 2018 at 7:57 AM Viktor Somogyi-Vass <
> viktorsomo...@gmail.com>
> wrote:
>
> > Thanks guys, I've updated my KIP with this info (so to keep solution #1).
> > If you find it good enough, please vote as well or let me know if you
> think
> > something is missing.
> >
> > On Sat, Aug 25, 2018 at 1:14 AM Ismael Juma  wrote:
> >
> > > I'm OK with 1 too. It makes me a bit sad that we don't have a path for
> > > removing the method without headers, but it seems like the simplest and
> > > least confusing option (I am assuming that headers are not needed in
> the
> > > serializers in the common case).
> > >
> > > Ismael
> > >
> > > On Fri, Aug 24, 2018 at 2:42 PM Jason Gustafson 
> > > wrote:
> > >
> > > > Hey Viktor,
> > > >
> > > > Good summary. I agree that option 1) seems like the simplest choice
> > and,
> > > as
> > > > you note, we can always add the default implementation later. I'll
> > leave
> > > > Ismael to make a case for the circular forwarding approach ;)
> > > >
> > > > -Jason
> > > >
> > > > On Fri, Aug 24, 2018 at 3:02 AM, Viktor Somogyi-Vass <
> > > > viktorsomo...@gmail.com> wrote:
> > > >
> > > > > I think in the first draft I didn't provide an implementation for
> > them
> > > as
> > > > > it seemed very simple and straightforward. I looked up a couple of
> > > > > implementations of the ExtendedSerializers on github and the
> general
> > > > > behavior seems to be that they delegate to the 2 argument
> > (headerless)
> > > > > method:
> > > > >
> > > > > https://github.com/khoitnm/practice-spring-kafka-grpc/blob/
> > > > > a6fc9b3395762c4889807baedd822f4653d5dcdd/kafka-common/src/
> > > > > main/java/org/tnmk/common/kafka/serialization/protobuf/
> > > > > ProtobufSerializer.java
> > > > >
> > https://github.com/hong-zhu/nxgen/blob/5cf1427d4e1a8f5c7fab47955af32a
> > > > > 0d4f4873af/nxgen-kafka-client/src/main/java/nxgen/kafka/
> > > > > client/event/serdes/EventSerializer.java
> > > > > https://github.com/jerry-jx/spring-kafka/blob/
> > > > > ac323ec5b8b9a0ca975db2f7322ff6878fce481a/spring-kafka/src/
> > > > >
> > > >
> > >
> >
> main/java/org/springframework/kafka/support/serializer/JsonSerializer.java
> > > > > https://github.com/enzobonggio/nonblocking-kafka/blob/
> > > > > bc1a379b2d9593b46cf9604063bc5b38e2785d19/src/main/java/com/
> > > > > example/kafka/producer/CustomJsonSerializer.java
> > > > >
> > > > > Of course 4 example is not representative but it shows that these
> > users
> > > > > usually delegate to the "headerless" (2 argument) method. I've
> tried
> > to
> > > > > look it up on other code search sites but haven't had much luck so
> > far.
> > > > > Given these examples and the way they implement them I'd say it's
> > more
> > > > > common to delegate to the headerless method, that's why I think
> it's
> > a
> > > > good
> > > > > approach for us too. Now having a default implementation for that
> is
> > > > again
> > > > > a good question. I think current use cases wouldn't change in
> either
> > > case
> > > > > (unless we deprecate the headerless one).
> > > > > For the new use cases it depends what do we want to propagate going
> > > > > forward. Do we want only one method to exist or two? As Ismael
> > > > highlighted
> > > > > it might be confusing if we have 2 methods, both with default
> > > > > implementation and in this case we want to push the 3 argument one
> > for
> > > > > users.
> > > > >
> > > > > So I see three possible ways:
> > > > > 1.) Don't provide a default implementation for the headerless
> method.
> > > > This
> > > > > supports the current implementations and encourages the delegation
> > > style
> > > > in
> > > > > future implementations. This might be the simplest option.
> > > > > 2.) Provide a default implementation for the headerless method.
> This
> > > > would
> > > > > be a bit confusing, so we'd likely push the use of the 3 parameter
> > > method
> > > > > and deprecate the headerless. This would however further litter the
> > > code
> > > > > base with deprecation warnings as we're using the headerless method
> > in
> > > a
> > > > > lot of places (think of the current serializers/deserializers). So
> in
> > > > this
> > > > > case we would want to clean up the code base a little where we can
> > and
> > > > may
> > > > > remove the headerless method entirely in Kafka 3. But they would
> hang
> > > > > around until that point. I think in this case the implementation
> for
> > > the
> > > > > headerless is a detail question as that is deprecated so we don't
> > > expect
> > > > > new implementations to use that method.
> > > > > If we decide to move this way, we have

[jira] [Resolved] (KAFKA-2385) zookeeper-shell does not work



 [ 
https://issues.apache.org/jira/browse/KAFKA-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2385.
--
Resolution: Fixed

Fixed in new zookeeper (3.4.7 +)  versions

> zookeeper-shell does not work
> -
>
> Key: KAFKA-2385
> URL: https://issues.apache.org/jira/browse/KAFKA-2385
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Jiangjie Qin
>Assignee: Flavio Junqueira
>Priority: Major
>
> The zookeeper shell shipped with Kafka does not work because jline jar is 
> missing.
> [jqin@jqin-ld1 bin]$ ./zookeeper-shell.sh localhost:2181
> Connecting to localhost:2181
> Welcome to ZooKeeper!
> JLine support is disabled
> WATCHER::
> WatchedEvent state:SyncConnected type:None path:null



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-2238) KafkaMetricsConfig cannot be configured in broker (KafkaConfig)