Jenkins build is back to normal : kafka-trunk-jdk8 #2447

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk8 #2446

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-6593; Fix livelock with consumer heartbeat thread in commitSync

--
[...truncated 3.50 MB...]

kafka.server.KafkaApisTest > 
shouldThrowUnsupportedVersionExceptionOnHandleTxnOffsetCommitRequestWhenInterBrokerProtocolNotSupported
 PASSED

kafka.server.KafkaApisTest > 
shouldThrowUnsupportedVersionExceptionOnHandleAddPartitionsToTxnRequestWhenInterBrokerProtocolNotSupported
 STARTED

kafka.server.KafkaApisTest > 
shouldThrowUnsupportedVersionExceptionOnHandleAddPartitionsToTxnRequestWhenInterBrokerProtocolNotSupported
 PASSED

kafka.server.KafkaApisTest > testReadUncommittedConsumerListOffsetLatest STARTED

kafka.server.KafkaApisTest > testReadUncommittedConsumerListOffsetLatest PASSED

kafka.server.KafkaApisTest > 
testMetadataRequestOnDistinctListenerWithInconsistentListenersAcrossBrokers 
STARTED

kafka.server.KafkaApisTest > 
testMetadataRequestOnDistinctListenerWithInconsistentListenersAcrossBrokers 
PASSED

kafka.server.KafkaApisTest > 
shouldAppendToLogOnWriteTxnMarkersWhenCorrectMagicVersion STARTED

kafka.server.KafkaApisTest > 
shouldAppendToLogOnWriteTxnMarkersWhenCorrectMagicVersion PASSED

kafka.server.KafkaApisTest > 
shouldThrowUnsupportedVersionExceptionOnHandleWriteTxnMarkersRequestWhenInterBrokerProtocolNotSupported
 STARTED

kafka.server.KafkaApisTest > 
shouldThrowUnsupportedVersionExceptionOnHandleWriteTxnMarkersRequestWhenInterBrokerProtocolNotSupported
 PASSED

kafka.server.KafkaApisTest > 
shouldRespondWithUnknownTopicWhenPartitionIsNotHosted STARTED

kafka.server.KafkaApisTest > 
shouldRespondWithUnknownTopicWhenPartitionIsNotHosted PASSED

kafka.server.KafkaApisTest > 
testReadCommittedConsumerListOffsetEarliestOffsetEqualsLastStableOffset STARTED

kafka.server.KafkaApisTest > 
testReadCommittedConsumerListOffsetEarliestOffsetEqualsLastStableOffset PASSED

kafka.server.KafkaApisTest > testReadCommittedConsumerListOffsetLatest STARTED

kafka.server.KafkaApisTest > testReadCommittedConsumerListOffsetLatest PASSED

kafka.server.KafkaApisTest > 
testReadCommittedConsumerListOffsetLimitedAtLastStableOffset STARTED

kafka.server.KafkaApisTest > 
testReadCommittedConsumerListOffsetLimitedAtLastStableOffset PASSED

kafka.server.KafkaApisTest > 
testMetadataRequestOnSharedListenerWithInconsistentListenersAcrossBrokers 
STARTED

kafka.server.KafkaApisTest > 
testMetadataRequestOnSharedListenerWithInconsistentListenersAcrossBrokers PASSED

kafka.server.KafkaApisTest > 
testReadUncommittedConsumerListOffsetEarliestOffsetEqualsHighWatermark STARTED

kafka.server.KafkaApisTest > 
testReadUncommittedConsumerListOffsetEarliestOffsetEqualsHighWatermark PASSED

kafka.server.KafkaApisTest > 
testReadUncommittedConsumerListOffsetLimitedAtHighWatermark STARTED

kafka.server.KafkaApisTest > 
testReadUncommittedConsumerListOffsetLimitedAtHighWatermark PASSED

kafka.server.KafkaApisTest > 
shouldThrowUnsupportedVersionExceptionOnHandleAddOffsetToTxnRequestWhenInterBrokerProtocolNotSupported
 STARTED

kafka.server.KafkaApisTest > 
shouldThrowUnsupportedVersionExceptionOnHandleAddOffsetToTxnRequestWhenInterBrokerProtocolNotSupported
 PASSED

kafka.server.KafkaApisTest > 
shouldRespondWithUnsupportedMessageFormatForBadPartitionAndNoErrorsForGoodPartition
 STARTED

kafka.server.KafkaApisTest > 
shouldRespondWithUnsupportedMessageFormatForBadPartitionAndNoErrorsForGoodPartition
 PASSED

kafka.server.KafkaApisTest > 
shouldThrowUnsupportedVersionExceptionOnHandleEndTxnRequestWhenInterBrokerProtocolNotSupported
 STARTED

kafka.server.KafkaApisTest > 
shouldThrowUnsupportedVersionExceptionOnHandleEndTxnRequestWhenInterBrokerProtocolNotSupported
 PASSED

kafka.server.KafkaApisTest > 
shouldRespondWithUnknownTopicOrPartitionForBadPartitionAndNoErrorsForGoodPartition
 STARTED

kafka.server.KafkaApisTest > 
shouldRespondWithUnknownTopicOrPartitionForBadPartitionAndNoErrorsForGoodPartition
 PASSED

kafka.server.DelegationTokenRequestsTest > testDelegationTokenRequests STARTED

kafka.server.DelegationTokenRequestsTest > testDelegationTokenRequests PASSED

kafka.server.ReplicaFetcherThreadFatalErrorTest > 
testFatalErrorInProcessFetchRequest STARTED

kafka.server.ReplicaFetcherThreadFatalErrorTest > 
testFatalErrorInProcessFetchRequest PASSED

kafka.server.ReplicaFetcherThreadFatalErrorTest > testFatalErrorInAddPartitions 
STARTED

kafka.server.ReplicaFetcherThreadFatalErrorTest > testFatalErrorInAddPartitions 
PASSED

kafka.server.ThrottledResponseExpirationTest > testThrottledRequest STARTED

kafka.server.ThrottledResponseExpirationTest > testThrottledRequest PASSED

kafka.server.ThrottledResponseExpirationTest > testExpire STARTED

kafka.server.ThrottledResponseExpirationTest > testExpire PASSED

kafka.server.ReplicationQuotaManagerTest > shouldThrottleOnlyDefinedReplicas 
STARTED

kafka.server.ReplicationQuotaManagerTest > shouldThrottleOnlyDefinedReplicas 
PASSED

答复: [DISCUSS] KIP-265: Make Windowed Serde to public APIs

[Hu Xi]

Guozhang,


Thanks for this KIP. Please help confirm questions below:

  1.   Do we also need to retrofit `SimpleConsumerShell` to have it support 
these newly-added serdes?
  2.   Does this KIP cover the changes for ConsoleConsumer as well?



发件人: Guozhang Wang 
发送时间: 2018年3月2日 8:20
收件人: dev@kafka.apache.org
主题: [DISCUSS] KIP-265: Make Windowed Serde to public APIs

Hello all,

I'd like to have a discussion on making windowed serde to public APIs of Kafka 
Streams. It involves a couple of new configs, plus a few new public classes for 
windowed serializer and deserializer, and also adding the corresponding console 
consumer options in order to fetch from a topic written by a windowed store.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-265%3A+Make+Windowed+Serde+to+public+APIs

I'd love to hear from your opinions on the proposed APIs, and if you have 
already encountered this and have to implement your own serdes, does the 
current public API fit your needs.


Thanks,

-- Guozhang

[jira] [Resolved] (KAFKA-6592) NullPointerException thrown when executing ConsoleCosumer with deserializer set to `WindowedDeserializer`

[huxihx (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

huxihx resolved KAFKA-6592.
---
Resolution: Duplicate

Seems it's a duplicate of 
[KAFKA-4831|https://issues.apache.org/jira/browse/KAFKA-4831]

> NullPointerException thrown when executing ConsoleCosumer with deserializer 
> set to `WindowedDeserializer`
> -
>
> Key: KAFKA-6592
> URL: https://issues.apache.org/jira/browse/KAFKA-6592
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 1.0.0
>Reporter: huxihx
>Assignee: huxihx
>Priority: Minor
>
> When reading streams app's output topic with WindowedDeserializer deserilizer 
> using kafka-console-consumer.sh, NullPointerException was thrown due to the 
> fact that the inner deserializer was not initialized since there is no place 
> in ConsoleConsumer to set this class.
> Complete stack trace is shown below:
> {code:java}
> [2018-02-26 14:56:04,736] ERROR Unknown error when running consumer:  
> (kafka.tools.ConsoleConsumer$)
> java.lang.NullPointerException
> at 
> org.apache.kafka.streams.kstream.internals.WindowedDeserializer.deserialize(WindowedDeserializer.java:89)
> at 
> org.apache.kafka.streams.kstream.internals.WindowedDeserializer.deserialize(WindowedDeserializer.java:35)
> at 
> kafka.tools.DefaultMessageFormatter.$anonfun$writeTo$2(ConsoleConsumer.scala:544)
> at scala.Option.map(Option.scala:146)
> at kafka.tools.DefaultMessageFormatter.write$1(ConsoleConsumer.scala:545)
> at kafka.tools.DefaultMessageFormatter.writeTo(ConsoleConsumer.scala:560)
> at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:147)
> at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:84)
> at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)
> at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [kafka-clients] Re: [VOTE] 1.1.0 RC0

[Jun Rao]

KAFKA-6111 is now merged to 1.1 branch.

Thanks,

Jun

On Thu, Mar 1, 2018 at 2:50 PM, Jun Rao  wrote:

> Hi, Damian,
>
> It would also be useful to include KAFKA-6111, which prevents
> deleteLogDirEventNotifications path to be deleted correctly from
> Zookeeper. The patch should be committed later today.
>
> Thanks,
>
> Jun
>
> On Thu, Mar 1, 2018 at 1:47 PM, Damian Guy  wrote:
>
>> Thanks Jason. Assuming the system tests pass i'll cut RC1 tomorrow.
>>
>> Thanks,
>> Damian
>>
>> On Thu, 1 Mar 2018 at 19:10 Jason Gustafson  wrote:
>>
>>> The fix has been merged to 1.1.
>>>
>>> Thanks,
>>> Jason
>>>
>>> On Wed, Feb 28, 2018 at 11:35 AM, Damian Guy 
>>> wrote:
>>>
>>> > Hi Jason,
>>> >
>>> > Ok - thanks. Let me know how you get on.
>>> >
>>> > Cheers,
>>> > Damian
>>> >
>>> > On Wed, 28 Feb 2018 at 19:23 Jason Gustafson 
>>> wrote:
>>> >
>>> > > Hey Damian,
>>> > >
>>> > > I think we should consider
>>> > > https://issues.apache.org/jira/browse/KAFKA-6593
>>> > > for the release. I have a patch available, but still working on
>>> > validating
>>> > > both the bug and the fix.
>>> > >
>>> > > -Jason
>>> > >
>>> > > On Wed, Feb 28, 2018 at 9:34 AM, Matthias J. Sax <
>>> matth...@confluent.io>
>>> > > wrote:
>>> > >
>>> > > > No. Both will be released.
>>> > > >
>>> > > > -Matthias
>>> > > >
>>> > > > On 2/28/18 6:32 AM, Marina Popova wrote:
>>> > > > > Sorry, maybe a stupid question, but:
>>> > > > >  I see that Kafka 1.0.1 RC2 is still not released, but now 1.1.0
>>> RC0
>>> > is
>>> > > > coming up...
>>> > > > > Does it mean 1.0.1 will be abandoned and we should be looking
>>> forward
>>> > > to
>>> > > > 1.1.0 instead?
>>> > > > >
>>> > > > > thanks!
>>> > > > >
>>> > > > > ​Sent with ProtonMail Secure Email.​
>>> > > > >
>>> > > > > ‐‐‐ Original Message ‐‐‐
>>> > > > >
>>> > > > > On February 26, 2018 6:28 PM, Vahid S Hashemian <
>>> > > > vahidhashem...@us.ibm.com> wrote:
>>> > > > >
>>> > > > >> +1 (non-binding)
>>> > > > >>
>>> > > > >> Built the source and ran quickstart (including streams)
>>> successfully
>>> > > on
>>> > > > >>
>>> > > > >> Ubuntu (with both Java 8 and Java 9).
>>> > > > >>
>>> > > > >> I understand the Windows platform is not officially supported,
>>> but I
>>> > > ran
>>> > > > >>
>>> > > > >> the same on Windows 10, and except for Step 7 (Connect)
>>> everything
>>> > > else
>>> > > > >>
>>> > > > >> worked fine.
>>> > > > >>
>>> > > > >> There are a number of warning and errors (including
>>> > > > >>
>>> > > > >> java.lang.ClassNotFoundException). Here's the final error
>>> message:
>>> > > > >>
>>> > > > >>> bin\\windows\\connect-standalone.bat
>>> config\\connect-standalone.
>>> > > > properties
>>> > > > >>
>>> > > > >> config\\connect-file-source.properties
>>> config\\connect-file-sink.
>>> > > > properties
>>> > > > >>
>>> > > > >> ...
>>> > > > >>
>>> > > > >> \[2018-02-26 14:55:56,529\] ERROR Stopping after connector error
>>> > > > >>
>>> > > > >> (org.apache.kafka.connect.cli.ConnectStandalone)
>>> > > > >>
>>> > > > >> java.lang.NoClassDefFoundError:
>>> > > > >>
>>> > > > >> org/apache/kafka/connect/transforms/util/RegexValidator
>>> > > > >>
>>> > > > >> at
>>> > > > >>
>>> > > > >> org.apache.kafka.connect.runtime.SinkConnectorConfig.<
>>> > > > clinit>(SinkConnectorConfig.java:46)
>>> > > > >>
>>> > > > >> at
>>> > > > >>
>>> > > > >>
>>> > > > >> org.apache.kafka.connect.runtime.AbstractHerder.
>>> > > > validateConnectorConfig(AbstractHerder.java:263)
>>> > > > >>
>>> > > > >> at
>>> > > > >>
>>> > > > >> org.apache.kafka.connect.runtime.standalone.StandaloneHerder.
>>> > > > putConnectorConfig(StandaloneHerder.java:164)
>>> > > > >>
>>> > > > >> at
>>> > > > >>
>>> > > > >> org.apache.kafka.connect.cli.ConnectStandalone.main(
>>> > > > ConnectStandalone.java:107)
>>> > > > >>
>>> > > > >> Caused by: java.lang.ClassNotFoundException:
>>> > > > >>
>>> > > > >> org.apache.kafka.connect.transforms.util.RegexValidator
>>> > > > >>
>>> > > > >> at
>>> > > > >>
>>> > > > >> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(
>>> > > > BuiltinClassLoader.java:582)
>>> > > > >>
>>> > > > >> at
>>> > > > >>
>>> > > > >> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.
>>> > > > loadClass(ClassLoaders.java:185)
>>> > > > >>
>>> > > > >> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:
>>> 496)
>>> > > > >>
>>> > > > >> ... 4 more
>>> > > > >>
>>> > > > >> Thanks for running the release.
>>> > > > >>
>>> > > > >> --Vahid
>>> > > > >>
>>> > > > >> From: Damian Guy damian@gmail.com
>>> > > > >>
>>> > > > >> To: dev@kafka.apache.org, us...@kafka.apache.org,
>>> > > > >>
>>> > > > >> kafka-clie...@googlegroups.com
>>> > > > >>
>>> > > > >> Date: 02/24/2018 08:16 AM
>>> > > > >>
>>> > > > >> Subject: \[VOTE\] 1.1.0 RC0
>>> > > > >>
>>> > > > >> Hello Kafka users, developers and client-developers,
>>> > > > >>
>>> > > > >> 

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

[Dong Lin]

Hey Jason,

Thanks much for all the valuable feedback!

On Wed, Feb 28, 2018 at 11:09 AM, Jason Gustafson 
wrote:

> Hi Dong,
>
> Great work on this proposal! Just a couple initial comments:
>
> My understanding is that the consumer will block on a topic until the all
> partitions have reached a certain partition epoch. What are the
> implications if a partition is offline? If we observe an epoch change while
> a partition is offline, it seems like we'd have to wait until the partition
> is back online before we can begin consuming the new epoch. Otherwise we
> will violate the ordering guarantees. Many use cases involve unordered
> data, so this would be a kind of regression in behavior, wouldn't it? A
> couple ideas:
>
> 1. Maybe we could have a topic configuration that controls whether or not
> ordering on the topic needs to be strictly followed? If we don't care about
> ordering, the consumer need not synchronize on epoch boundaries and we need
> not care about offline partitions.
>

Very good point. Though Kafka admin should make sure that offline partition
happens really rarely (with proper RF etc.), I agree that it is bad to let
partitions of non-keyed topic blocking waiting for an offline partition.

I have updated the KIP to include enable.ordered.delivery as a new
topic-level config. Since offline partitions should happen very rarely and
most keyed users would want to get the ordered guarantee by default, I have
set this config to true by default. I don't have a strong opinion on the
default value though.



> 2. Waiting on all partitions allows for any key partitioning function. It's
> good because it's general, but it is overly conservative when the
> partitioning function has finer control over key movement. For example, if
> the partitioner only allows for splits, then there is just one partition to
> await before consuming a new epoch for any given partition. I am not sure
> what it would look like, but I'm wondering if it would be possible to
> leverage the custom partitioning logic on the consumer side as well to
> avoid unneeded waiting.


Yeah I have thought about this. That is why I originally wanted to always
double the partition number of a topic. For example, if we double partition
number from 6 to 12, we know that consumer only needs to wait for partition
1 before consuming the first message in partition 7.

I think we may be able to optimize the consumer-side performance while
still allowing partition expansion to arbitrary number (e.g. not limited by
double), by using the Linear Hashing algorithm that Jay suggested. I will
think more about it.


>
> I think piggybacking the epoch exchanges onto the consumer heartbeats is a
> good idea. Just wanted to mention that consumers are not the only ones
> using the heartbeat API. For example, Kafka Connect also uses the group
> protocol to balance its load. Of course other use cases could leave these
> fields empty, but it's a little odd to have the protocol tailored
> specifically for one use case. To be honest, the group management protocol
> is one of the messier Kafka APIs and I don't think anyone is satisfied with
> the current approach. We need not redesign the whole thing in this KIP, but
> it might be nice to consider some options so that we're sure we're either
> heading in a better direction or at least not making things more confusing
> than they already are. The challenge is that it's useful to have some
> coordinator logic specific to the group type. I can imagine down the road
> that other use cases may also have some custom metadata which they need to
> piggyback on the heartbeat and they may also need the coordinator to do
> some facilitation. Maybe the heartbeat protocol could be left generic and
> we could have a separate module in the GroupCoordinator for custom consumer
> logic? Not too sure the best way to go.
>

Good point. I wasn't aware that HeartbeatRequest and HeartbeatResponse are
also used outside consumer. Since these are used for other purposes, I have
instead created ConsumerGroupPositionRequest and
ConsumerGroupPositionResponse as suggested.



>
> Thanks,
> Jason
>
>
> On Tue, Feb 27, 2018 at 11:49 PM, Stephane Maarek <
> steph...@simplemachines.com.au> wrote:
>
> > Sounds awesome !
> > Are you planning to have auto scaling of partitions in a following KIP ?
> > That would be the holy grail
> >
> > On 28 Feb. 2018 5:13 pm, "Dong Lin"  wrote:
> >
> > > Hey Jan,
> > >
> > > I am not sure if it is acceptable for producer to be stopped for a
> while,
> > > particularly for online application which requires low latency. I am
> also
> > > not sure how consumers can switch to a new topic. Does user application
> > > needs to explicitly specify a different topic for producer/consumer to
> > > subscribe to? It will be helpful for discussion if you can provide more
> > > detail on the interface change for this solution.
> > >
> > > Thanks,
> > > Dong
> > >
> > > On Mon, Feb 

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

[Allen Wang]

Hi Dong,



On Tue, Feb 27, 2018 at 10:07 PM, Dong Lin  wrote:

> Hey Allen,
>
> Thanks for the comments.
>
> On Mon, Feb 26, 2018 at 9:27 PM, Allen Wang 
> wrote:
>
> > Hi Dong,
> >
> > Please see my comments inline.
> >
> > Thanks,
> > Allen
> >
> > On Sun, Feb 25, 2018 at 3:33 PM, Dong Lin  wrote:
> >
> > > Hey Allen,
> > >
> > > Thanks for your comment. I will comment inline.
> > >
> > > On Thu, Feb 22, 2018 at 3:05 PM, Allen Wang  >
> > > wrote:
> > >
> > > > Overall this is a very useful feature. With this we can finally scale
> > > keyed
> > > > messages.
> > > >
> > > > +1 on the ability to remove partitions. This will greatly increase
> > > Kafka's
> > > > scalability in cloud.
> > > >
> > > > For example, when there is traffic increase, we can add brokers and
> > > assign
> > > > new partitions to the new brokers. When traffic decreases, we can
> mark
> > > > these new partitions as read only and remove them afterwards,
> together
> > > with
> > > > the brokers that host these partitions. This will be a light-weight
> > > > approach to scale a Kafka cluster compared to partition reassignment
> > > where
> > > > you will always have to move data.
> > > >
> > > > I have some suggestions:
> > > >
> > > > - The KIP described each step in detail which is great. However, it
> > lacks
> > > > the "why" part to explain the high level goal we want to achieve with
> > > each
> > > > step. For example, the purpose of step 5 may be described as "Make
> sure
> > > > consumers always first finish consuming all data prior to partition
> > > > expansion to enforce message ordering".
> > > >
> > >
> > > Yeah I think this is useful. This is a non-trivial KIP and it is useful
> > to
> > > explain the motivation of each change to help reading. I will added
> > > motivation for each change in the KIP. Please let me know if there is
> > > anything else that can make the KIP more readable.
> > >
> > >
> > > >
> > > > - The rejection of produce request at partition expansion should be
> > > > configurable because it does not matter for non-keyed messages. Same
> > with
> > > > the consumer behavior for step 5. This will ensure that for non-keyed
> > > > messages, partition expansion does not add the cost of possible
> message
> > > > drop on producer or message latency on the consumer.
> > > >
> > >
> > > Ideally we would like to avoid adding extra configs to keep the
> interface
> > > simple. I think the current overhead in the producer is actually very
> > > small. Partition expansion or deletion should happen very infrequently.
> > > Note that our producer today needs to refresh metadata whenever there
> is
> > > leadership movement, i.e. producer will receive
> > > NotLeaderForPartitionException from the old leader and keep refreshing
> > > metadata until it gets the new leader of the partition, which happens
> > much
> > > more frequently than Partition expansion or deletion. So I am not sure
> we
> > > should add a config to optimize this.
> > >
> >
> > I was concerned that at high message rate, rejecting requests could lead
> to
> > producer side buffer full and lead to unnecessary message drop on
> producer
> > side for non-keyed messages.
> >
> > What about the delay on consumer? It could be significant when one
> consumer
> > is lagging for certain partitions and all consumers in the same group
> have
> > to wait. This delay could be significant and again unnecessary for
> messages
> > where the order does not matter.
> >
>
> I agree this may increase delay on the producer side. Consumer is not
> impacted directly and any extra delay in consumer all comes from the extra
> delay in producer.
>
> Note that when broker has leadership change, if producer's metadata is
> still using the old leader, producer will see
> NotLeaderForPartitionException and will have to repeatedly update
> metadadata until the metadata uses the new leader. Do you think the
> metadata update after partition expansion, as introduced in this KIP, is
> any worse than the metadata update required during leadership change? If
> not, given that our user already needs to handle or tolerate the extra
> delay during leadership change, I think the extra delay after partition
> expansion should be fine.
>

I think the difference here is that when you expand partitions, almost all
produce requests will be rejected at that time. So this will have a
significant impact on producer buffer. Usually the leadership changes only
affect a few partitions so the impact is small.

On the consumer side, as Jason pointed out, all consumers will have to
synchronize on leader epoch. Let's say a consumer is lagging behind in a
partition which exists prior to expansion, and it takes an hour to catch
up. During this hour, all consumers in the same group will have to wait.
This is completely waste of time if ordering is not important.



>

> >
> >
> > >
> > >
> > >
> > > 

[DISCUSS] KIP-265: Make Windowed Serde to public APIs

[Guozhang Wang]

Hello all,

I'd like to have a discussion on making windowed serde to public APIs of
Kafka Streams. It involves a couple of new configs, plus a few new public
classes for windowed serializer and deserializer, and also adding the
corresponding console consumer options in order to fetch from a topic
written by a windowed store.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-265%3A+Make+Windowed+Serde+to+public+APIs

I'd love to hear from your opinions on the proposed APIs, and if you have
already encountered this and have to implement your own serdes, does the
current public API fit your needs.


Thanks,

-- Guozhang

[jira] [Resolved] (KAFKA-3513) Transient failure of OffsetValidationTest

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-3513.

Resolution: Fixed

We haven't seen this in some time, so I'm going to close this. We can reopen if 
it reoccurs.

> Transient failure of OffsetValidationTest
> -
>
> Key: KAFKA-3513
> URL: https://issues.apache.org/jira/browse/KAFKA-3513
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, system tests
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
>Priority: Major
>
> http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2016-04-05--001.1459840046--apache--trunk--31e263e/report.html
> The version of the test fails in this case is:
> Module: kafkatest.tests.client.consumer_test
> Class:  OffsetValidationTest
> Method: test_broker_failure
> Arguments:
> {
>   "clean_shutdown": true,
>   "enable_autocommit": false
> }
> and others passed. It's unclear if the parameters actually have any impact on 
> the failure.
> I did some initial triage and it looks like the test code isn't seeing all 
> the group members join the group (receive partition assignments), but it 
> appears from the logs that they all did. This could indicate a simple timing 
> issue, but I haven't been able to verify that yet.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [kafka-clients] Re: [VOTE] 1.1.0 RC0

[Jun Rao]

Hi, Damian,

It would also be useful to include KAFKA-6111, which prevents
deleteLogDirEventNotifications
path to be deleted correctly from Zookeeper. The patch should be committed
later today.

Thanks,

Jun

On Thu, Mar 1, 2018 at 1:47 PM, Damian Guy  wrote:

> Thanks Jason. Assuming the system tests pass i'll cut RC1 tomorrow.
>
> Thanks,
> Damian
>
> On Thu, 1 Mar 2018 at 19:10 Jason Gustafson  wrote:
>
>> The fix has been merged to 1.1.
>>
>> Thanks,
>> Jason
>>
>> On Wed, Feb 28, 2018 at 11:35 AM, Damian Guy 
>> wrote:
>>
>> > Hi Jason,
>> >
>> > Ok - thanks. Let me know how you get on.
>> >
>> > Cheers,
>> > Damian
>> >
>> > On Wed, 28 Feb 2018 at 19:23 Jason Gustafson 
>> wrote:
>> >
>> > > Hey Damian,
>> > >
>> > > I think we should consider
>> > > https://issues.apache.org/jira/browse/KAFKA-6593
>> > > for the release. I have a patch available, but still working on
>> > validating
>> > > both the bug and the fix.
>> > >
>> > > -Jason
>> > >
>> > > On Wed, Feb 28, 2018 at 9:34 AM, Matthias J. Sax <
>> matth...@confluent.io>
>> > > wrote:
>> > >
>> > > > No. Both will be released.
>> > > >
>> > > > -Matthias
>> > > >
>> > > > On 2/28/18 6:32 AM, Marina Popova wrote:
>> > > > > Sorry, maybe a stupid question, but:
>> > > > >  I see that Kafka 1.0.1 RC2 is still not released, but now 1.1.0
>> RC0
>> > is
>> > > > coming up...
>> > > > > Does it mean 1.0.1 will be abandoned and we should be looking
>> forward
>> > > to
>> > > > 1.1.0 instead?
>> > > > >
>> > > > > thanks!
>> > > > >
>> > > > > ​Sent with ProtonMail Secure Email.​
>> > > > >
>> > > > > ‐‐‐ Original Message ‐‐‐
>> > > > >
>> > > > > On February 26, 2018 6:28 PM, Vahid S Hashemian <
>> > > > vahidhashem...@us.ibm.com> wrote:
>> > > > >
>> > > > >> +1 (non-binding)
>> > > > >>
>> > > > >> Built the source and ran quickstart (including streams)
>> successfully
>> > > on
>> > > > >>
>> > > > >> Ubuntu (with both Java 8 and Java 9).
>> > > > >>
>> > > > >> I understand the Windows platform is not officially supported,
>> but I
>> > > ran
>> > > > >>
>> > > > >> the same on Windows 10, and except for Step 7 (Connect)
>> everything
>> > > else
>> > > > >>
>> > > > >> worked fine.
>> > > > >>
>> > > > >> There are a number of warning and errors (including
>> > > > >>
>> > > > >> java.lang.ClassNotFoundException). Here's the final error
>> message:
>> > > > >>
>> > > > >>> bin\\windows\\connect-standalone.bat
>> config\\connect-standalone.
>> > > > properties
>> > > > >>
>> > > > >> config\\connect-file-source.properties
>> config\\connect-file-sink.
>> > > > properties
>> > > > >>
>> > > > >> ...
>> > > > >>
>> > > > >> \[2018-02-26 14:55:56,529\] ERROR Stopping after connector error
>> > > > >>
>> > > > >> (org.apache.kafka.connect.cli.ConnectStandalone)
>> > > > >>
>> > > > >> java.lang.NoClassDefFoundError:
>> > > > >>
>> > > > >> org/apache/kafka/connect/transforms/util/RegexValidator
>> > > > >>
>> > > > >> at
>> > > > >>
>> > > > >> org.apache.kafka.connect.runtime.SinkConnectorConfig.<
>> > > > clinit>(SinkConnectorConfig.java:46)
>> > > > >>
>> > > > >> at
>> > > > >>
>> > > > >>
>> > > > >> org.apache.kafka.connect.runtime.AbstractHerder.
>> > > > validateConnectorConfig(AbstractHerder.java:263)
>> > > > >>
>> > > > >> at
>> > > > >>
>> > > > >> org.apache.kafka.connect.runtime.standalone.StandaloneHerder.
>> > > > putConnectorConfig(StandaloneHerder.java:164)
>> > > > >>
>> > > > >> at
>> > > > >>
>> > > > >> org.apache.kafka.connect.cli.ConnectStandalone.main(
>> > > > ConnectStandalone.java:107)
>> > > > >>
>> > > > >> Caused by: java.lang.ClassNotFoundException:
>> > > > >>
>> > > > >> org.apache.kafka.connect.transforms.util.RegexValidator
>> > > > >>
>> > > > >> at
>> > > > >>
>> > > > >> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(
>> > > > BuiltinClassLoader.java:582)
>> > > > >>
>> > > > >> at
>> > > > >>
>> > > > >> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.
>> > > > loadClass(ClassLoaders.java:185)
>> > > > >>
>> > > > >> at java.base/java.lang.ClassLoader.loadClass(
>> ClassLoader.java:496)
>> > > > >>
>> > > > >> ... 4 more
>> > > > >>
>> > > > >> Thanks for running the release.
>> > > > >>
>> > > > >> --Vahid
>> > > > >>
>> > > > >> From: Damian Guy damian@gmail.com
>> > > > >>
>> > > > >> To: dev@kafka.apache.org, us...@kafka.apache.org,
>> > > > >>
>> > > > >> kafka-clie...@googlegroups.com
>> > > > >>
>> > > > >> Date: 02/24/2018 08:16 AM
>> > > > >>
>> > > > >> Subject: \[VOTE\] 1.1.0 RC0
>> > > > >>
>> > > > >> Hello Kafka users, developers and client-developers,
>> > > > >>
>> > > > >> This is the first candidate for release of Apache Kafka 1.1.0.
>> > > > >>
>> > > > >> This is minor version release of Apache Kakfa. It Includes 29 new
>> > > KIPs.
>> > > > >>
>> > > > >> Please see the release plan for more details:
>> > > > >>
>> > > > >> 

[jira] [Created] (KAFKA-6605) Flatten SMT does not properly handle fields that are null

[Randall Hauch (JIRA)]

Randall Hauch created KAFKA-6605:


 Summary: Flatten SMT does not properly handle fields that are null
 Key: KAFKA-6605
 URL: https://issues.apache.org/jira/browse/KAFKA-6605
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 1.0.0
Reporter: Randall Hauch


When a message has a null field, the `Flatten` SMT does not properly handle 
this and throws an NPE. Consider this message from Debezium:

{code}
{
  "before": null,
  "after": {
"dbserver1.mydb.team.Value": {
  "id": 1,
  "name": "kafka",
  "email": "ka...@apache.org",
  "last_modified": 1519939449000
}
  },
  "source": {
"version": {
  "string": "0.7.3"
},
"name": "dbserver1",
"server_id": 0,
"ts_sec": 0,
"gtid": null,
"file": "mysql-bin.03",
"pos": 154,
"row": 0,
"snapshot": {
  "boolean": true
},
"thread": null,
"db": {
  "string": "mydb"
},
"table": {
  "string": "team"
}
  },
  "op": "c",
  "ts_ms": {
"long": 1519939520285
  }
}
{code}

Note how `before` is null; this event represents a row was INSERTED and thus 
there is no `before` state of the row. This results in an NPE:

{noformat}
org.apache.avro.SchemaParseException: Illegal character in: source.version
at org.apache.avro.Schema.validateName(Schema.java:1151)
at org.apache.avro.Schema.access$200(Schema.java:81)
at org.apache.avro.Schema$Field.(Schema.java:403)
at 
org.apache.avro.SchemaBuilder$FieldBuilder.completeField(SchemaBuilder.java:2124)
at 
org.apache.avro.SchemaBuilder$FieldBuilder.completeField(SchemaBuilder.java:2116)
at 
org.apache.avro.SchemaBuilder$FieldBuilder.access$5300(SchemaBuilder.java:2034)
at 
org.apache.avro.SchemaBuilder$GenericDefault.withDefault(SchemaBuilder.java:2423)
at 
io.confluent.connect.avro.AvroData.addAvroRecordField(AvroData.java:898)
at 
io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:799)
at 
io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:652)
at 
io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:647)
at io.confluent.connect.avro.AvroData.fromConnectData(AvroData.java:324)
at 
io.confluent.connect.avro.AvroConverter.fromConnectData(AvroConverter.java:75)
at 
org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:220)
at 
org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:187)
at 
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}

Here's the connector configuration that was used:

{code}
{
"name": "debezium-connector-flatten",
"config": {
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"tasks.max": "1",
"database.hostname": "mysql",
"database.port": "3306",
"database.user": "debezium",
"database.password": "dbz",
"database.server.id": "223345",
"database.server.name": "dbserver-flatten",
"database.whitelist": "mydb",
"database.history.kafka.bootstrap.servers": 
"kafka-1:9092,kafka-2:9092,kafka-3:9092",
"database.history.kafka.topic": "schema-flatten.mydb",
"include.schema.changes": "true",
"transforms": "flatten",
"transforms.flatten.type": 
"org.apache.kafka.connect.transforms.Flatten$Value"
  }
}
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk8 #2445

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-6560: Replace range query with newly added single point query in

--
[...truncated 3.92 MB...]

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testRegisterPersistentStore STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testRegisterPersistentStore PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldSuccessfullyReInitializeStateStores STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldSuccessfullyReInitializeStateStores PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldThrowIllegalArgumentExceptionOnRegisterWhenStoreHasAlreadyBeenRegistered 
STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldThrowIllegalArgumentExceptionOnRegisterWhenStoreHasAlreadyBeenRegistered 
PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldRestoreStoreWithSinglePutRestoreSpecification STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldRestoreStoreWithSinglePutRestoreSpecification PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldNotChangeOffsetsIfAckedOffsetsIsNull STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldNotChangeOffsetsIfAckedOffsetsIsNull PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldThrowIllegalArgumentExceptionIfStoreNameIsSameAsCheckpointFileName STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
shouldThrowIllegalArgumentExceptionIfStoreNameIsSameAsCheckpointFileName PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldCloseStateManagerWithOffsets STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldCloseStateManagerWithOffsets PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldProcessRecordsForTopic STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldProcessRecordsForTopic PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeProcessorTopology STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeProcessorTopology PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeContext STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeContext PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldCheckpointOffsetsWhenStateIsFlushed STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldCheckpointOffsetsWhenStateIsFlushed PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldThrowStreamsExceptionWhenKeyDeserializationFails STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldThrowStreamsExceptionWhenKeyDeserializationFails PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeStateManager STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldInitializeStateManager PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldProcessRecordsForOtherTopic STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldProcessRecordsForOtherTopic PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldNotThrowStreamsExceptionWhenValueDeserializationFails STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldNotThrowStreamsExceptionWhenValueDeserializationFails PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldThrowStreamsExceptionWhenValueDeserializationFails STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldThrowStreamsExceptionWhenValueDeserializationFails PASSED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldNotThrowStreamsExceptionWhenKeyDeserializationFailsWithSkipHandler STARTED

org.apache.kafka.streams.processor.internals.GlobalStateTaskTest > 
shouldNotThrowStreamsExceptionWhenKeyDeserializationFailsWithSkipHandler PASSED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testThroughputMetrics STARTED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testThroughputMetrics PASSED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testLatencyMetrics STARTED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testLatencyMetrics PASSED

org.apache.kafka.streams.processor.internals.StreamsMetricsImplTest > 
testRemoveSensor 

Re: [VOTE] 1.1.0 RC0

[Damian Guy]

Thanks Jason. Assuming the system tests pass i'll cut RC1 tomorrow.

Thanks,
Damian

On Thu, 1 Mar 2018 at 19:10 Jason Gustafson  wrote:

> The fix has been merged to 1.1.
>
> Thanks,
> Jason
>
> On Wed, Feb 28, 2018 at 11:35 AM, Damian Guy  wrote:
>
> > Hi Jason,
> >
> > Ok - thanks. Let me know how you get on.
> >
> > Cheers,
> > Damian
> >
> > On Wed, 28 Feb 2018 at 19:23 Jason Gustafson  wrote:
> >
> > > Hey Damian,
> > >
> > > I think we should consider
> > > https://issues.apache.org/jira/browse/KAFKA-6593
> > > for the release. I have a patch available, but still working on
> > validating
> > > both the bug and the fix.
> > >
> > > -Jason
> > >
> > > On Wed, Feb 28, 2018 at 9:34 AM, Matthias J. Sax <
> matth...@confluent.io>
> > > wrote:
> > >
> > > > No. Both will be released.
> > > >
> > > > -Matthias
> > > >
> > > > On 2/28/18 6:32 AM, Marina Popova wrote:
> > > > > Sorry, maybe a stupid question, but:
> > > > >  I see that Kafka 1.0.1 RC2 is still not released, but now 1.1.0
> RC0
> > is
> > > > coming up...
> > > > > Does it mean 1.0.1 will be abandoned and we should be looking
> forward
> > > to
> > > > 1.1.0 instead?
> > > > >
> > > > > thanks!
> > > > >
> > > > > ​Sent with ProtonMail Secure Email.​
> > > > >
> > > > > ‐‐‐ Original Message ‐‐‐
> > > > >
> > > > > On February 26, 2018 6:28 PM, Vahid S Hashemian <
> > > > vahidhashem...@us.ibm.com> wrote:
> > > > >
> > > > >> +1 (non-binding)
> > > > >>
> > > > >> Built the source and ran quickstart (including streams)
> successfully
> > > on
> > > > >>
> > > > >> Ubuntu (with both Java 8 and Java 9).
> > > > >>
> > > > >> I understand the Windows platform is not officially supported,
> but I
> > > ran
> > > > >>
> > > > >> the same on Windows 10, and except for Step 7 (Connect) everything
> > > else
> > > > >>
> > > > >> worked fine.
> > > > >>
> > > > >> There are a number of warning and errors (including
> > > > >>
> > > > >> java.lang.ClassNotFoundException). Here's the final error message:
> > > > >>
> > > > >>> bin\\windows\\connect-standalone.bat config\\connect-standalone.
> > > > properties
> > > > >>
> > > > >> config\\connect-file-source.properties config\\connect-file-sink.
> > > > properties
> > > > >>
> > > > >> ...
> > > > >>
> > > > >> \[2018-02-26 14:55:56,529\] ERROR Stopping after connector error
> > > > >>
> > > > >> (org.apache.kafka.connect.cli.ConnectStandalone)
> > > > >>
> > > > >> java.lang.NoClassDefFoundError:
> > > > >>
> > > > >> org/apache/kafka/connect/transforms/util/RegexValidator
> > > > >>
> > > > >> at
> > > > >>
> > > > >> org.apache.kafka.connect.runtime.SinkConnectorConfig.<
> > > > clinit>(SinkConnectorConfig.java:46)
> > > > >>
> > > > >> at
> > > > >>
> > > > >>
> > > > >> org.apache.kafka.connect.runtime.AbstractHerder.
> > > > validateConnectorConfig(AbstractHerder.java:263)
> > > > >>
> > > > >> at
> > > > >>
> > > > >> org.apache.kafka.connect.runtime.standalone.StandaloneHerder.
> > > > putConnectorConfig(StandaloneHerder.java:164)
> > > > >>
> > > > >> at
> > > > >>
> > > > >> org.apache.kafka.connect.cli.ConnectStandalone.main(
> > > > ConnectStandalone.java:107)
> > > > >>
> > > > >> Caused by: java.lang.ClassNotFoundException:
> > > > >>
> > > > >> org.apache.kafka.connect.transforms.util.RegexValidator
> > > > >>
> > > > >> at
> > > > >>
> > > > >> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(
> > > > BuiltinClassLoader.java:582)
> > > > >>
> > > > >> at
> > > > >>
> > > > >> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.
> > > > loadClass(ClassLoaders.java:185)
> > > > >>
> > > > >> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)
> > > > >>
> > > > >> ... 4 more
> > > > >>
> > > > >> Thanks for running the release.
> > > > >>
> > > > >> --Vahid
> > > > >>
> > > > >> From: Damian Guy damian@gmail.com
> > > > >>
> > > > >> To: dev@kafka.apache.org, us...@kafka.apache.org,
> > > > >>
> > > > >> kafka-clie...@googlegroups.com
> > > > >>
> > > > >> Date: 02/24/2018 08:16 AM
> > > > >>
> > > > >> Subject: \[VOTE\] 1.1.0 RC0
> > > > >>
> > > > >> Hello Kafka users, developers and client-developers,
> > > > >>
> > > > >> This is the first candidate for release of Apache Kafka 1.1.0.
> > > > >>
> > > > >> This is minor version release of Apache Kakfa. It Includes 29 new
> > > KIPs.
> > > > >>
> > > > >> Please see the release plan for more details:
> > > > >>
> > > > >> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
> > > > apache.org_confluence_pages_viewpage.action-3FpageId-
> > > > 3D71764913=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_
> > > >
> > > itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=
> > K9Iz2hWA2pj4QGxW6fleW20K0M7oEe
> > > > WCbqs5nbbUY0c=M1liORvtcIt7pZ8e5GnLr9a1i6SOUY4bvjHYOrY_zcE=
> > > > >>
> > > > >> A few highlights:
> > > > >>
> > > > >> -   Significant Controller improvements (much faster and session
> > > > expiration
> > > > >>
> > > > >> edge 

[jira] [Created] (KAFKA-6604) ReplicaManager should not remove partitions on the log dirctory from high watermark checkpoint file

[Dong Lin (JIRA)]

Dong Lin created KAFKA-6604:
---

 Summary: ReplicaManager should not remove partitions on the log 
dirctory from high watermark checkpoint file
 Key: KAFKA-6604
 URL: https://issues.apache.org/jira/browse/KAFKA-6604
 Project: Kafka
  Issue Type: Bug
Reporter: Dong Lin
Assignee: Dong Lin


Currently a broker may truncate a partition to log start offset in the 
following scenario:

- Broker A is restarted after shutdown
- Controller knows that broker A is started.
- Som event (e.g. topic deletion) triggered controller to send 
LeaderAndIsrRequest for partition P1.
- Broker A receives LeaderAndIsrRequest for partition P1. After the broker 
receives the first LeaderAndIsrRequest, it will overwrite the HW checkpoint 
file with all its leader partitions and follower partitions. The checkpoint 
file will contain only the HW for partition P1.
- Controller sends broker A a LeaderAndIsrRequest for all its leader and 
follower partitions.
- Broker creates ReplicaFetcherThread for its follower partitions, truncates 
the log to HW, which will be zero for all partitions except P1.

When this happens, potentially all logs in the broker will be truncated to log 
start offset and then the cluster will run with reduced availability for a long 
time.

The right solution is to keep the partitions in the high watermark checkpoint 
file if the partition exists in LogManager.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

[Vahid S Hashemian]

Correction.

The offsets will be removed when the expiration timestamp is reached.
We will be setting the expiration timestamp in a smart manner to cover 
both the issue reported in the JIRA, and also the case of unsubscribed 
topics.

I'll detail this in the KIP, and send a notification when the update is 
ready for review.

Apologies for the confusion.
--Vahid




From:   "Vahid S Hashemian" 
To: dev@kafka.apache.org
Date:   03/01/2018 11:43 AM
Subject:Re: [DISCUSS] KIP-211: Revise Expiration Semantics of 
Consumer Group Offsets



Hi Jason,

Thanks for your feedback.

If we want to keep the per-partition expiration timestamp, then the scope 
of the KIP will be significantly reduced. In other words, we will just be 
deleting offsets from cache as before but only if the group is in Empty 
state.
This would not cause any negative backward compatibility concern, since 
offsets won't expire any earlier than before. They may be removed at the 
same time (if group is already Empty) or later (if group is not Empty yet) 

than before.
If I'm not mistaken we no longer need to change the internal schema either 

- no need for keeping track of when group becomes Empty.
Would this make sense? If so, I'll update the KIP with this reduced-scope 
proposal.

Regarding manual offset removal I was thinking of synching up the new 
consumer group command with the old command in which a '--topic' option 
was supported with '--delete'.

Regarding the other JIRA you referred to, sure, I'll add that in the KIP.

Thanks.
--Vahid



From:   Jason Gustafson 
To: dev@kafka.apache.org
Date:   02/28/2018 12:10 PM
Subject:Re: [DISCUSS] KIP-211: Revise Expiration Semantics of 
Consumer Group Offsets



Hey Vahid,

Thanks for the response. Replies below:


> 1. I think my suggestion in the KIP was more towards ignoring the client
> provided values and use a large enough broker config value instead. It
> seems the question comes down to whether we still want to honor the
> `retention_time` field in the old requests. With the new request (as per
> this KIP) the client would not be able to overwrite the broker retention
> config. Your suggestion provides kind of a back door for the overwrite.
> Also, since different offset commits associated with a group can
> potentially use different `retention_time` values, it's probably
> reasonable to use the maximum of all those values (including the broker
> config) as the group offset retention.


Mainly I wanted to ensure that we would be holding offsets at least as 
long
as what was requested by older clients. If we hold it for longer, that's
probably fine, but there may be application behavior which would break if
offsets are expired earlier than expected.

2. If I'm not mistake you are referring to potential changes in
> `GROUP_METADATA_VALUE_SCHEMA`. I saw this as an internal implementation
> matter and frankly, have not fully thought about it, but I agree that it
> needs to be updated to include either the timestamp the group becomes
> `Empty` or maybe the expiration timestamp of the group. And perhaps, we
> would not need to store per partition offset expiration timestamp 
anymore.
> Is there a particular reason for your suggestion of storing the 
timestamp
> the group becomes `Empty`, vs the expiration timestamp of the group?


Although it is not exposed to clients, we still have to manage
compatibility of the schema across versions, so I think we should include
it in the KIP. The reason I was thinking of using the time that the group
became Empty is that the configured timeout might change. I think my
expectation as a user would be that a timeout change would also apply to
existing groups, but I'm not sure if there are any reasons not to so.

3. To limit the scope of the KIP I would prefer to handle this matter
> separately if it doesn't have to be addressed as part of this change. It
> probably needs be addressed at some point and I'll mention it in the KIP
> so we have it documented. Do you think my suggestion of manually 
removing
> topic offsets from group (as an interim solution) is worth additional
> discussion / implementation?


I think manual removal of offsets for this case is a bit of a tough sell
for usability. Did you imagine it happening automatically in the consumer
through an API?

I'm finding it increasingly frustrating that the generic group coordinator
is limited in its decision making since it cannot see the subscription
metadata. It is the same problem in Dong's KIP. I think I would suggest
that, at a minimum, we leave the door open to enforcing offset expiration
either 1) when the group becomes empty, and 2) when the corresponding
partition is removed from the subscription. Perhaps that means we need to
keep the individual offset expiration timestamp after all. Actually we
would probably need it anyway to handle "simple" consumer groups which are
always Empty.

One additional note: I have seen 

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

[Vahid S Hashemian]

Hi Jason,

Thanks for your feedback.

If we want to keep the per-partition expiration timestamp, then the scope 
of the KIP will be significantly reduced. In other words, we will just be 
deleting offsets from cache as before but only if the group is in Empty 
state.
This would not cause any negative backward compatibility concern, since 
offsets won't expire any earlier than before. They may be removed at the 
same time (if group is already Empty) or later (if group is not Empty yet) 
than before.
If I'm not mistaken we no longer need to change the internal schema either 
- no need for keeping track of when group becomes Empty.
Would this make sense? If so, I'll update the KIP with this reduced-scope 
proposal.

Regarding manual offset removal I was thinking of synching up the new 
consumer group command with the old command in which a '--topic' option 
was supported with '--delete'.

Regarding the other JIRA you referred to, sure, I'll add that in the KIP.

Thanks.
--Vahid



From:   Jason Gustafson 
To: dev@kafka.apache.org
Date:   02/28/2018 12:10 PM
Subject:Re: [DISCUSS] KIP-211: Revise Expiration Semantics of 
Consumer Group Offsets



Hey Vahid,

Thanks for the response. Replies below:


> 1. I think my suggestion in the KIP was more towards ignoring the client
> provided values and use a large enough broker config value instead. It
> seems the question comes down to whether we still want to honor the
> `retention_time` field in the old requests. With the new request (as per
> this KIP) the client would not be able to overwrite the broker retention
> config. Your suggestion provides kind of a back door for the overwrite.
> Also, since different offset commits associated with a group can
> potentially use different `retention_time` values, it's probably
> reasonable to use the maximum of all those values (including the broker
> config) as the group offset retention.


Mainly I wanted to ensure that we would be holding offsets at least as 
long
as what was requested by older clients. If we hold it for longer, that's
probably fine, but there may be application behavior which would break if
offsets are expired earlier than expected.

2. If I'm not mistake you are referring to potential changes in
> `GROUP_METADATA_VALUE_SCHEMA`. I saw this as an internal implementation
> matter and frankly, have not fully thought about it, but I agree that it
> needs to be updated to include either the timestamp the group becomes
> `Empty` or maybe the expiration timestamp of the group. And perhaps, we
> would not need to store per partition offset expiration timestamp 
anymore.
> Is there a particular reason for your suggestion of storing the 
timestamp
> the group becomes `Empty`, vs the expiration timestamp of the group?


Although it is not exposed to clients, we still have to manage
compatibility of the schema across versions, so I think we should include
it in the KIP. The reason I was thinking of using the time that the group
became Empty is that the configured timeout might change. I think my
expectation as a user would be that a timeout change would also apply to
existing groups, but I'm not sure if there are any reasons not to so.

3. To limit the scope of the KIP I would prefer to handle this matter
> separately if it doesn't have to be addressed as part of this change. It
> probably needs be addressed at some point and I'll mention it in the KIP
> so we have it documented. Do you think my suggestion of manually 
removing
> topic offsets from group (as an interim solution) is worth additional
> discussion / implementation?


I think manual removal of offsets for this case is a bit of a tough sell
for usability. Did you imagine it happening automatically in the consumer
through an API?

I'm finding it increasingly frustrating that the generic group coordinator
is limited in its decision making since it cannot see the subscription
metadata. It is the same problem in Dong's KIP. I think I would suggest
that, at a minimum, we leave the door open to enforcing offset expiration
either 1) when the group becomes empty, and 2) when the corresponding
partition is removed from the subscription. Perhaps that means we need to
keep the individual offset expiration timestamp after all. Actually we
would probably need it anyway to handle "simple" consumer groups which are
always Empty.

One additional note: I have seen recently a case where the offset cache
caused an OOM on the broker. I looked into it and found that most of the
cache was used for storing console consumer offsets. I know you had a 
patch
before which turned off auto-commit when the groupId was generated by
ConsoleConsumer. Maybe we could lump that change into this KIP?

Thanks,
Jason




On Fri, Feb 23, 2018 at 4:08 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi Jason,
>
> Thanks a lot for reviewing the KIP.
>
> 1. I think my suggestion in the KIP was more towards ignoring the client
> provided values and use 

[jira] [Created] (KAFKA-6603) Kafka streams off heap memory usage does not match expected values from configuration

[Igor Calabria (JIRA)]

Igor Calabria created KAFKA-6603:


 Summary: Kafka streams off heap memory usage does not match 
expected values from configuration
 Key: KAFKA-6603
 URL: https://issues.apache.org/jira/browse/KAFKA-6603
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.0.0
Reporter: Igor Calabria


Hi, I have a simple aggregation pipeline that's backed by the default state 
store(rocksdb). The pipeline works fine except that off heap the memory usage 
is way higher than expected. Following the 
[documention|https://docs.confluent.io/current/streams/developer-guide/config-streams.html#streams-developer-guide-rocksdb-config]
 has some effect(memory usage is reduced) but the values don't match at all. 

The java process is set to run with just `-Xmx300m -Xms300m`  and rocksdb 
config looks like this
{code:java}
tableConfig.setCacheIndexAndFilterBlocks(true);
tableConfig.setBlockCacheSize(1048576); //1MB
tableConfig.setBlockSize(16 * 1024); // 16KB
options.setTableFormatConfig(tableConfig);
options.setMaxWriteBufferNumber(2);
options.setWriteBufferSize(8 * 1024); // 8KB{code}
To estimate memory usage, I'm using this formula  
{noformat}
(block_cache_size + write_buffer_size * write_buffer_number) * segments * 
partitions{noformat}
Since my topic has 25 partitions with 3 segments each(it's a windowed store), 
off heap memory usage should be about 76MB. What I'm seeing in production is 
upwards of 300MB, even taking in consideration  extra overhead from rocksdb 
compaction threads, this seems a bit high (especially when the disk usage for 
all files is just 1GB) 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : kafka-trunk-jdk7 #3220

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-6593) Coordinator disconnect in heartbeat thread can cause commitSync to block indefinitely

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-6593.

Resolution: Fixed

> Coordinator disconnect in heartbeat thread can cause commitSync to block 
> indefinitely
> -
>
> Key: KAFKA-6593
> URL: https://issues.apache.org/jira/browse/KAFKA-6593
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 1.0.0, 0.11.0.2
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Critical
> Fix For: 1.1.0
>
> Attachments: consumer.log
>
>
> If a coordinator disconnect is observed in the heartbeat thread, it can cause 
> a pending offset commit to be cancelled just before the foreground thread 
> begins waiting on its response in poll(). Since the poll timeout is 
> Long.MAX_VALUE, this will cause the consumer to effectively hang until some 
> other network event causes the poll() to return. We try to protect this case 
> with a poll condition on the future, but this isn't bulletproof since the 
> future can be completed outside of the lock.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] 1.1.0 RC0

[Jason Gustafson]

The fix has been merged to 1.1.

Thanks,
Jason

On Wed, Feb 28, 2018 at 11:35 AM, Damian Guy  wrote:

> Hi Jason,
>
> Ok - thanks. Let me know how you get on.
>
> Cheers,
> Damian
>
> On Wed, 28 Feb 2018 at 19:23 Jason Gustafson  wrote:
>
> > Hey Damian,
> >
> > I think we should consider
> > https://issues.apache.org/jira/browse/KAFKA-6593
> > for the release. I have a patch available, but still working on
> validating
> > both the bug and the fix.
> >
> > -Jason
> >
> > On Wed, Feb 28, 2018 at 9:34 AM, Matthias J. Sax 
> > wrote:
> >
> > > No. Both will be released.
> > >
> > > -Matthias
> > >
> > > On 2/28/18 6:32 AM, Marina Popova wrote:
> > > > Sorry, maybe a stupid question, but:
> > > >  I see that Kafka 1.0.1 RC2 is still not released, but now 1.1.0 RC0
> is
> > > coming up...
> > > > Does it mean 1.0.1 will be abandoned and we should be looking forward
> > to
> > > 1.1.0 instead?
> > > >
> > > > thanks!
> > > >
> > > > ​Sent with ProtonMail Secure Email.​
> > > >
> > > > ‐‐‐ Original Message ‐‐‐
> > > >
> > > > On February 26, 2018 6:28 PM, Vahid S Hashemian <
> > > vahidhashem...@us.ibm.com> wrote:
> > > >
> > > >> +1 (non-binding)
> > > >>
> > > >> Built the source and ran quickstart (including streams) successfully
> > on
> > > >>
> > > >> Ubuntu (with both Java 8 and Java 9).
> > > >>
> > > >> I understand the Windows platform is not officially supported, but I
> > ran
> > > >>
> > > >> the same on Windows 10, and except for Step 7 (Connect) everything
> > else
> > > >>
> > > >> worked fine.
> > > >>
> > > >> There are a number of warning and errors (including
> > > >>
> > > >> java.lang.ClassNotFoundException). Here's the final error message:
> > > >>
> > > >>> bin\\windows\\connect-standalone.bat config\\connect-standalone.
> > > properties
> > > >>
> > > >> config\\connect-file-source.properties config\\connect-file-sink.
> > > properties
> > > >>
> > > >> ...
> > > >>
> > > >> \[2018-02-26 14:55:56,529\] ERROR Stopping after connector error
> > > >>
> > > >> (org.apache.kafka.connect.cli.ConnectStandalone)
> > > >>
> > > >> java.lang.NoClassDefFoundError:
> > > >>
> > > >> org/apache/kafka/connect/transforms/util/RegexValidator
> > > >>
> > > >> at
> > > >>
> > > >> org.apache.kafka.connect.runtime.SinkConnectorConfig.<
> > > clinit>(SinkConnectorConfig.java:46)
> > > >>
> > > >> at
> > > >>
> > > >>
> > > >> org.apache.kafka.connect.runtime.AbstractHerder.
> > > validateConnectorConfig(AbstractHerder.java:263)
> > > >>
> > > >> at
> > > >>
> > > >> org.apache.kafka.connect.runtime.standalone.StandaloneHerder.
> > > putConnectorConfig(StandaloneHerder.java:164)
> > > >>
> > > >> at
> > > >>
> > > >> org.apache.kafka.connect.cli.ConnectStandalone.main(
> > > ConnectStandalone.java:107)
> > > >>
> > > >> Caused by: java.lang.ClassNotFoundException:
> > > >>
> > > >> org.apache.kafka.connect.transforms.util.RegexValidator
> > > >>
> > > >> at
> > > >>
> > > >> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(
> > > BuiltinClassLoader.java:582)
> > > >>
> > > >> at
> > > >>
> > > >> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.
> > > loadClass(ClassLoaders.java:185)
> > > >>
> > > >> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)
> > > >>
> > > >> ... 4 more
> > > >>
> > > >> Thanks for running the release.
> > > >>
> > > >> --Vahid
> > > >>
> > > >> From: Damian Guy damian@gmail.com
> > > >>
> > > >> To: dev@kafka.apache.org, us...@kafka.apache.org,
> > > >>
> > > >> kafka-clie...@googlegroups.com
> > > >>
> > > >> Date: 02/24/2018 08:16 AM
> > > >>
> > > >> Subject: \[VOTE\] 1.1.0 RC0
> > > >>
> > > >> Hello Kafka users, developers and client-developers,
> > > >>
> > > >> This is the first candidate for release of Apache Kafka 1.1.0.
> > > >>
> > > >> This is minor version release of Apache Kakfa. It Includes 29 new
> > KIPs.
> > > >>
> > > >> Please see the release plan for more details:
> > > >>
> > > >> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
> > > apache.org_confluence_pages_viewpage.action-3FpageId-
> > > 3D71764913=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg=Q_
> > >
> > itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc=
> K9Iz2hWA2pj4QGxW6fleW20K0M7oEe
> > > WCbqs5nbbUY0c=M1liORvtcIt7pZ8e5GnLr9a1i6SOUY4bvjHYOrY_zcE=
> > > >>
> > > >> A few highlights:
> > > >>
> > > >> -   Significant Controller improvements (much faster and session
> > > expiration
> > > >>
> > > >> edge cases fixed)
> > > >>
> > > >> -   Data balancing across log directories (JBOD)
> > > >> -   More efficient replication when the number of partitions is
> large
> > > >> -   Dynamic Broker Configs
> > > >> -   Delegation tokens (KIP-48)
> > > >> -   Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)
> > > >>
> > > >> Release notes for the 1.1.0 release:
> > > >>
> > > >> https://urldefense.proofpoint.com/v2/url?u=http-3A__home.
> > > 

[jira] [Created] (KAFKA-6602) Support Kafka to save credentials in Java Key Store on Zookeeper node

[Chen He (JIRA)]

Chen He created KAFKA-6602:
--

 Summary: Support Kafka to save credentials in Java Key Store on 
Zookeeper node
 Key: KAFKA-6602
 URL: https://issues.apache.org/jira/browse/KAFKA-6602
 Project: Kafka
  Issue Type: New Feature
  Components: security
Reporter: Chen He


Kafka connect needs to talk to multifarious distributed systems. However, each 
system has its own authentication mechanism. How we manage these credentials 
become a common problem. 

Here are my thoughts:
 # We may need to save it in java key store;
 # We may need to put this key store in a distributed system (topic or 
zookeeper);
 # Key store password may be configured in Kafka configuration;

I have implement the feature that allows store java key store in zookeeper 
node. If Kafka community likes this idea, I am happy to contribute.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6560) Use single-point queries than range queries for windowed aggregation operators

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-6560.
--
   Resolution: Fixed
Fix Version/s: 1.2.0

> Use single-point queries than range queries for windowed aggregation operators
> --
>
> Key: KAFKA-6560
> URL: https://issues.apache.org/jira/browse/KAFKA-6560
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>Priority: Critical
>  Labels: needs-kip
> Fix For: 1.2.0
>
>
> Today for windowed aggregations in Streams DSL, the underlying implementation 
> is leveraging the fetch(key, from, to) API to get all the related windows for 
> a single record to update. However, this is a very inefficient operation with 
> significant amount of CPU time iterating over window stores. On the other 
> hand, since the operator implementation itself have full knowledge of the 
> window specs it can actually translate this operation into multiple 
> single-point queries with the accurate window start timestamp, which would 
> largely reduce the overhead.
> The proposed approach is to add a single fetch API to the WindowedStore and 
> use that in the KStreamWindowedAggregate / KStreamWindowedReduce operators.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-261: Add Single Value Fetch in Window Stores

[Guozhang Wang]

+1 from myself.

I'm going to close the vote with the following results:

binding +1s: 3 (Matthias, Damian, Guozhang)
non-binding +1s: 2 (Ted, Bill)


Thanks for everyone who have participated in the discussion and voted!


Guozhang


On Sat, Feb 24, 2018 at 3:51 PM, Matthias J. Sax 
wrote:

> +1 (binding)
>
>
> On 2/24/18 12:25 PM, Bill Bejeck wrote:
> > Thanks for the KIP!
> >
> > +1
> >
> > Bill
> >
> > On Sat, Feb 24, 2018 at 2:33 PM, Damian Guy 
> wrote:
> >
> >> Thanks Guozhang!
> >> +1
> >>
> >> On Sat, 24 Feb 2018 at 19:12 Ted Yu  wrote:
> >>
> >>> +1
> >>>
> >>> On Sat, Feb 24, 2018 at 11:10 AM, Guozhang Wang 
> >>> wrote:
> >>>
>  Hi all,
> 
>  I want to start voting on KIP-261 to add a new API for window stores
> in
>  order to optimize our current windowed aggregation implementations
> >> inside
>  Streams DSL:
> 
>  https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>  261%3A+Add+Single+Value+Fetch+in+Window+Stores
> 
> 
>  Please cast your vote before next Wednesday, EOD.
> 
> 
>  Thanks,
>  -- Guozhang
> 
> >>>
> >>
> >
>
>


-- 
-- Guozhang

[jira] [Resolved] (KAFKA-6396) Possibly kafka-connect converter should be able to stop processing chain

[Alexander Koval (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Koval resolved KAFKA-6396.

Resolution: Invalid

Thank you for the explanation.

> Possibly kafka-connect converter should be able to stop processing chain
> 
>
> Key: KAFKA-6396
> URL: https://issues.apache.org/jira/browse/KAFKA-6396
> Project: Kafka
>  Issue Type: Wish
>  Components: KafkaConnect
>Affects Versions: 1.0.0
>Reporter: Alexander Koval
>Priority: Minor
>
> At present only transformations can discard records returning null. But I 
> think sometimes it would be nice to discard processing chain after converting 
> message. For example I have some tags shipped with a message key and I want 
> to stop processing the message after converting its key (there are a lot of 
> messages and I don't want to deserialize message values that I don't need).
> At the moment to do that I should disable converters and move message 
> deserializing to the transformation chain:
> {code}
> key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
> value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
> transforms=proto,catalog
> transforms.proto.type=company.evo.kafka.ProtobufTransformation
> transforms.proto.key.protobuf.class=company.evo.uaprom.indexator.KeyProto$KeyMessage
> transforms.proto.value.protobuf.class=company.evo.uaprom.indexator.catalog.CompanyProto$UniversalCompanyMessage
> transforms.proto.tag=catalog
> {code}
> If 
> [WorkerSinkTask|https://github.com/apache/kafka/blob/1.0.0/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L453]
>  checked converted values on {{null}} it would solved my problem more 
> gracefully



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)