Re: [DISCUSS] KIP-478 Strongly Typed Processor API

[Matthias J. Sax]

I think `RecordProcessor` is a good name.


-Matthias

On 6/21/19 5:09 PM, John Roesler wrote:
> After kicking the naming around a bit more, it seems like any package
> name change is a bit "weird" because it fragments the package and
> directory structure. If we can come up with a reasonable name for the
> interface after all, it seems like the better choice.
> 
> The real challenge is that the existing name "Processor" seems just
> about perfect. In picking a new name, we need to consider the ultimate
> state, after the deprecation period, when we entirely remove
> Processor. In this context, TypedProcessor seems a little odd to me,
> because it seems to imply that there should also be an "untyped
> processor".
> 
> After kicking around a few other ideas, what does everyone think about
> "RecordProcessor"? I _think_ maybe it stands on its own just fine,
> because it's a thing that processes... records?
> 
> If others agree with this, I can change the proposal to RecordProcessor.
> 
> Thanks,
> -John
> 
> On Fri, Jun 21, 2019 at 6:42 PM John Roesler  wrote:
>>
>> Hi all,
>>
>> I've updated the KIP with the feedback so far.
>>
>> The naming question is still the biggest (only?) outstanding issue. It
>> would be good to hear some more thoughts on it.
>>
>> As we stand now, there's one vote for changing the package name to
>> something like 'typedprocessor', one for changing the interface to
>> TypedProcessor (as in the PoC), and one for just changing the
>> Processor interface in-place, breaking source compatibility.
>>
>> How can we resolve this decision?
>>
>> Thanks,
>> -John
>>
>> On Thu, Jun 20, 2019 at 5:44 PM John Roesler  wrote:
>>>
>>> Thanks for the feedback, Guozhang and Matthias,
>>>
>>> Regarding motivation: I'll update the wiki. Briefly:
>>> * Any processor can benefit. Imagine a pure user of the ProcessorAPI
>>> who has very complex processing logic. I have seen several processor
>>> implementation that are hundreds of lines long and call
>>> `context.forward` in many different locations and branches. In such an
>>> implementation, it would be very easy to have a bug in a rarely used
>>> branch that forwards the wrong kind of value. This would structurally
>>> prevent that from happening.
>>> * Also, anyone who heavily uses the ProcessorAPI would likely have
>>> developed helper methods to wire together processors, just as we have
>>> in the DSL implementation. This change would enable them to ensure at
>>> compile time that they are actually wiring together compatible types.
>>> This was actually _my_ original motivation, since I found it very
>>> difficult and time consuming to follow the Streams DSL internal
>>> builders.
>>>
>>> Regarding breaking the source compatibility of Processor: I would
>>> _love_ to side-step the naming problem, but I really don't know if
>>> it's excusable to break compatibility. I suspect that our oldest and
>>> dearest friends are using the ProcessorAPI in some form or another,
>>> and all their source code would break. It sucks to have to create a
>>> whole new interface to get around this, but it feels like the right
>>> thing to do. Would be nice to get even more feedback on this point,
>>> though.
>>>
>>> Regarding the types of stores, as I said in my response to Sophie,
>>> it's not an issue.
>>>
>>> Regarding the change to StreamsBuilder, it doesn't pin the types in
>>> any way, since all the types are bounded by Object only, and there are
>>> no extra constraints between arguments (each type is used only once in
>>> one argument). But maybe I missed the point you were asking about.
>>> Since the type takes generic paramters, we should allow users to pass
>>> in parameterized arguments. Otherwise, they would _have to_ give us a
>>> raw type, and they would be forced to get a "rawtyes" warning from the
>>> compiler. So, it's our obligation in any API that accepts a
>>> parameterized-type parameter to allow people to actually pass a
>>> parameterized type, even if we don't actually use the parameters.
>>>
>>> The naming question is a complex one, as I took pains to detail
>>> previously. Please don't just pick out one minor point, call it weak,
>>> and then claim that it invalidates the whole decision. I don't think
>>> there's a clear best choice, so I'm more than happy for someone to
>>> advocate for renaming the class instead of the package. Can you
>>> provide some reasons why you think that would be better?
>>>
>>> Regarding the deprecated methods, you're absolutely right. I'll update the 
>>> KIP.
>>>
>>> Thanks again for all the feedback!
>>> -John
>>>
>>> On Thu, Jun 20, 2019 at 4:34 PM Matthias J. Sax  
>>> wrote:

 Just want to second what Sophie said about the stores. The type of a
 used stores is completely independent of input/output types.

 This related to change `addGlobalStore()` method. Why do you want to pin
 the types? In fact, people request the ability to filter() and maybe
 even map() the data before they 

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

[Matthias J. Sax]

Yes, something like this. I did not think about good configuration
parameter names yet. I am also not sure if I understand all proposed
configs atm. But all configs should be listed and explained in the KIP
anyway, and we can discuss further after you have updated the KIP (I can
ask more detailed question if I have any).


-Matthias

On 6/21/19 2:05 PM, Development wrote:
> Yes, you are right. ByteSerializer is not what I need to have in a list
> of primitives.
> 
> As for the default constructor and configurability, just want to make
> sure. Is this what you have on your mind?
> 
> Best,
> Daniyar Yeralin
> 
> 
> 
>> On Jun 21, 2019, at 2:51 PM, Matthias J. Sax > > wrote:
>>
>> Thanks for the update!
>>
>> I think that `ListDeserializer`, `ListSerializer`, and `ListSerde`
>> should have an default constructor and it should be possible to pass in
>> the `Class listClass` information via a configuration. Otherwise,
>> KafkaStreams cannot use it as default serde.
>>
>>
>> For the primitive serializers: `BytesSerializer` is not primitive IMHO,
>> as is it for `byte[]` with variable length -- it's for arrays, not for
>> single `byte` (note, that `Bytes` is a Kafka class wrapping `byte[]`).
>>
>>
>> For tests, we can comment on the PR. No need to do this in the KIP
>> discussion.
>>
>>
>> Can you also update the KIP?
>>
>>
>>
>> -Matthias
>>
>>
>>
>>
>>
>> On 6/21/19 11:29 AM, Development wrote:
>>> I made and pushed necessary commits, so we could review the final
>>> version under PR https://github.com/apache/kafka/pull/6592
>>>
>>> I also need some advice on writing tests for this new serde. So far I
>>> only have two test cases (roundtrip and empty payload), I’m not sure
>>> if it is enough.
>>>
>>> Thank y’all for your help in this KIP :)
>>>
>>> Best,
>>> Daniyar Yeralin
>>>
>>>
 On Jun 21, 2019, at 1:44 PM, John Roesler >>> > wrote:

 Hey Daniyar,

 Looks good to me! Thanks for considering it.

 Thanks,
 -John

 On Fri, Jun 21, 2019 at 9:04 AM Development >>>  > wrote:
 Hey John and Matthias,

 Yes, now I see it all. I’m storing lots of redundant information.
 Here is my final idea. Yes, now a user should pass a list type. I
 realized that’s the type is not really needed in ListSerializer, but
 only in ListDeserializer:


 In ListSerializer we will start storing sizes only if serializer is
 not a primitive serializer:


 Then, in deserializer, we persist passed list type, so that during
 deserialization we could create an instance of it with predefined
 listSize for better performance.
 We also try to locate a primitiveSize based on passed deserializer.
 If it is not there, then primitiveSize will be null. Which means
 that each entry’s size was encoded individually.


 This looks much cleaner and more concise.

 What do you think?

 Best,
 Daniyar Yeralin

> On Jun 20, 2019, at 5:45 PM, Matthias J. Sax   > wrote:
>
> For encoding the list-type: I see John's point about re-encoding the
> list-type redundantly. However, I also don't like the idea that the
> Deserializer returns a fixed type...
>
> Maybe it's best allow users to specify the target list type on
> deserialization via config?
>
> Similar for the primitive types: I don't think we need to encode the
> type size, but users could specify the type on the deserializer (via a
> config again)?
>
>
> About generics: nesting could be arbitrarily deep. Hence, I doubt
> we can
> support this and a cast will be necessary at some point in the user
> code.
>
>
>
> -Matthias
>
>
>
> On 6/20/19 1:21 PM, John Roesler wrote:
>> Hey Daniyar,
>>
>> Thanks for looking at it!
>>
>> Something like your screenshot is more along the lines of what I was
>> thinking. Sorry, but I didn't follow what you mean, how would that not
>> be "vanilla java"?
>>
>> Unfortunately the deserializer needs more information, though. For
>> example, what if the inner type is a Map? The serde
>> could
>> only be used to produce a LinkedList, thus, we'd still need an
>> inner serde, like you have in the KIP (Serde innerSerde).
>>
>> Something more like Serde> = Serdes.listSerde(
>>  /**list type**/ LinkedList.class,
>>  /**inner serde**/ new MyRecordSerde()
>> )
>>
>> And in configuration, it's something like:
>> default.key.serde: org...ListSerde
>> default.key.list.serde.type: java.util.LinkedList
>> default.key.list.serde.inner: com.mycompany.MyRecordSerde
>>
>>
>> What do you think?
>> Thanks,
>> -John
>>
>> On Thu, Jun 20, 2019 at 2:46 PM Development 

Build failed in Jenkins: kafka-trunk-jdk8 #3744

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Reflection free implementation of `defaultKerberosRealm` (#6978)

--
[...truncated 2.52 MB...]

org.apache.kafka.clients.InFlightRequestsTest > 
testCompleteLastSentThrowsIfNoInFlights PASSED

org.apache.kafka.clients.InFlightRequestsTest > testCompleteNext STARTED

org.apache.kafka.clients.InFlightRequestsTest > testCompleteNext PASSED

org.apache.kafka.clients.InFlightRequestsTest > testCompleteLastSent STARTED

org.apache.kafka.clients.InFlightRequestsTest > testCompleteLastSent PASSED

org.apache.kafka.clients.InFlightRequestsTest > testClearAll STARTED

org.apache.kafka.clients.InFlightRequestsTest > testClearAll PASSED

org.apache.kafka.clients.InFlightRequestsTest > testTimedOutNodes STARTED

org.apache.kafka.clients.InFlightRequestsTest > testTimedOutNodes PASSED

org.apache.kafka.clients.ApiVersionsTest > testMaxUsableProduceMagic STARTED

org.apache.kafka.clients.ApiVersionsTest > testMaxUsableProduceMagic PASSED

org.apache.kafka.clients.NetworkClientTest > testDefaultRequestTimeout STARTED

org.apache.kafka.clients.NetworkClientTest > testDefaultRequestTimeout PASSED

org.apache.kafka.clients.NetworkClientTest > testSimpleRequestResponse STARTED

org.apache.kafka.clients.NetworkClientTest > testSimpleRequestResponse PASSED

org.apache.kafka.clients.NetworkClientTest > 
testUnsupportedVersionDuringInternalMetadataRequest STARTED

org.apache.kafka.clients.NetworkClientTest > 
testUnsupportedVersionDuringInternalMetadataRequest PASSED

org.apache.kafka.clients.NetworkClientTest > 
testDisconnectDuringUserMetadataRequest STARTED

org.apache.kafka.clients.NetworkClientTest > 
testDisconnectDuringUserMetadataRequest PASSED

org.apache.kafka.clients.NetworkClientTest > testClose STARTED

org.apache.kafka.clients.NetworkClientTest > testClose PASSED

org.apache.kafka.clients.NetworkClientTest > 
testConnectionDelayDisconnectedWithNoExponentialBackoff STARTED

org.apache.kafka.clients.NetworkClientTest > 
testConnectionDelayDisconnectedWithNoExponentialBackoff PASSED

org.apache.kafka.clients.NetworkClientTest > testLeastLoadedNode STARTED

org.apache.kafka.clients.NetworkClientTest > testLeastLoadedNode PASSED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelayConnected 
STARTED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelayConnected PASSED

org.apache.kafka.clients.NetworkClientTest > testRequestTimeout STARTED

org.apache.kafka.clients.NetworkClientTest > testRequestTimeout PASSED

org.apache.kafka.clients.NetworkClientTest > 
testConnectionDelayConnectedWithNoExponentialBackoff STARTED

org.apache.kafka.clients.NetworkClientTest > 
testConnectionDelayConnectedWithNoExponentialBackoff PASSED

org.apache.kafka.clients.NetworkClientTest > 
testSimpleRequestResponseWithNoBrokerDiscovery STARTED

org.apache.kafka.clients.NetworkClientTest > 
testSimpleRequestResponseWithNoBrokerDiscovery PASSED

org.apache.kafka.clients.NetworkClientTest > 
testThrottlingNotEnabledForConnectionToOlderBroker STARTED

org.apache.kafka.clients.NetworkClientTest > 
testThrottlingNotEnabledForConnectionToOlderBroker PASSED

org.apache.kafka.clients.NetworkClientTest > 
testSimpleRequestResponseWithStaticNodes STARTED

org.apache.kafka.clients.NetworkClientTest > 
testSimpleRequestResponseWithStaticNodes PASSED

org.apache.kafka.clients.NetworkClientTest > 
testLeastLoadedNodeConsidersThrottledConnections STARTED

org.apache.kafka.clients.NetworkClientTest > 
testLeastLoadedNodeConsidersThrottledConnections PASSED

org.apache.kafka.clients.NetworkClientTest > 
testClientDisconnectAfterInternalApiVersionRequest STARTED

org.apache.kafka.clients.NetworkClientTest > 
testClientDisconnectAfterInternalApiVersionRequest PASSED

org.apache.kafka.clients.NetworkClientTest > testCallDisconnect STARTED

org.apache.kafka.clients.NetworkClientTest > testCallDisconnect PASSED

org.apache.kafka.clients.NetworkClientTest > testDnsLookupFailure STARTED

org.apache.kafka.clients.NetworkClientTest > testDnsLookupFailure PASSED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelay STARTED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelay PASSED

org.apache.kafka.clients.NetworkClientTest > testConnectionThrottling STARTED

org.apache.kafka.clients.NetworkClientTest > testConnectionThrottling PASSED

org.apache.kafka.clients.NetworkClientTest > 
testServerDisconnectAfterInternalApiVersionRequest STARTED

org.apache.kafka.clients.NetworkClientTest > 
testServerDisconnectAfterInternalApiVersionRequest PASSED

org.apache.kafka.clients.NetworkClientTest > 
testConnectionDelayWithNoExponentialBackoff STARTED

org.apache.kafka.clients.NetworkClientTest > 
testConnectionDelayWithNoExponentialBackoff PASSED

org.apache.kafka.clients.NetworkClientTest > testSendToUnreadyNode STARTED

org.apache.kafka.clients.NetworkClientTest > 

Build failed in Jenkins: kafka-trunk-jdk8 #3743

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: Fix DistributedHerderTest after adding reason to 
maybeLeaveGroup

[github] KAFKA-8106: Skipping ByteBuffer allocation of key / value / headers in

--
[...truncated 2.52 MB...]

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertEmptyListToListWithoutSchema STARTED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertEmptyListToListWithoutSchema PASSED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertStringWithQuotesAndOtherDelimiterCharacters STARTED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertStringWithQuotesAndOtherDelimiterCharacters PASSED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertListWithMixedValuesToListWithoutSchema STARTED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertListWithMixedValuesToListWithoutSchema PASSED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertMapWithStringKeys STARTED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertMapWithStringKeys PASSED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldParseStringOfMapWithStringValuesWithWhitespaceAsMap STARTED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldParseStringOfMapWithStringValuesWithWhitespaceAsMap PASSED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertMapWithStringKeysAndShortValues STARTED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertMapWithStringKeysAndShortValues PASSED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldParseStringOfMapWithStringValuesWithoutWhitespaceAsMap STARTED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldParseStringOfMapWithStringValuesWithoutWhitespaceAsMap PASSED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertSimpleString STARTED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertSimpleString PASSED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertEmptyMapToMapWithoutSchema STARTED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertEmptyMapToMapWithoutSchema PASSED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertEmptyString STARTED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertEmptyString PASSED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertMapWithStringKeysAndMixedValuesToMapWithoutSchema STARTED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertMapWithStringKeysAndMixedValuesToMapWithoutSchema PASSED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldParseStringOfMapWithIntValuesWithoutWhitespaceAsMap STARTED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldParseStringOfMapWithIntValuesWithoutWhitespaceAsMap PASSED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldParseStringOfMapWithShortValuesWithWhitespaceAsMap STARTED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldParseStringOfMapWithShortValuesWithWhitespaceAsMap PASSED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertMapWithStringKeysAndIntegerValues STARTED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertMapWithStringKeysAndIntegerValues PASSED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertNullValue STARTED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertNullValue PASSED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldParseStringOfMapWithShortValuesWithoutWhitespaceAsMap STARTED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldParseStringOfMapWithShortValuesWithoutWhitespaceAsMap PASSED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertListWithStringValues STARTED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertListWithStringValues PASSED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldParseStringOfMapWithIntValuesWithWhitespaceAsMap STARTED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldParseStringOfMapWithIntValuesWithWhitespaceAsMap PASSED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertListWithIntegerValues STARTED

org.apache.kafka.connect.storage.SimpleHeaderConverterTest > 
shouldConvertListWithIntegerValues PASSED

org.apache.kafka.connect.header.ConnectHeaderTest > shouldGetSchemaFromStruct 
STARTED

org.apache.kafka.connect.header.ConnectHeaderTest > shouldGetSchemaFromStruct 
PASSED

org.apache.kafka.connect.header.ConnectHeaderTest > 

Re: Preliminary blog post for the Apache Kafka 2.3.0 release

[Ismael Juma]

Thanks Colin! Maybe we should mention that restarts are much faster when
you have a lot of partitions:
 https://issues.apache.org/jira/browse/KAFKA-7283

"We did experiments on a cluster with 15 brokers, each of which has ~3k
segments (and there are 31.8k partitions with RF=3 which are evenly
distributed across brokers; total bytes-in-rate is around 400 MBps). The
results show that rolling bounce time reduces from 135 minutes to 55
minutes."

Ismael

On Tue, Jun 18, 2019 at 1:10 PM Colin McCabe  wrote:

> Hi all,
>
> I've written up a preliminary blog post about the upcoming Apache Kafka
> 2.3.0 release.  Take a look and let me know what you think.
>
>
> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache
>
> cheers,
> Colin
>

Re: Preliminary blog post for the Apache Kafka 2.3.0 release

[Konstantine Karantasis]

Looks great to me Colin. Thanks for the small adjustments!

Cheers,
Konstantine

On Thu, Jun 20, 2019 at 9:50 AM Colin McCabe  wrote:

> On Thu, Jun 20, 2019, at 00:23, Matthias J. Sax wrote:
> > Great blog post, Colin!
> >
> > Two comments:
> >
> >
> > (1) KIP-258: "future features" -> "the ability to return the latest
> > timestamp in Interactive Queries"
> >
> > This is not a future feature, but the timestamp can be queried in 2.3
> > already.
>
> Thanks for the correction.  This should be fixed.
>
> >
> >
> > (2) Why only listing KIP-428; KIP-445 is equally important.
> >
>
> Good point.  I added information about KIP-445.
>
> best,
> Colin
>
> >
> >
> > -Matthias
> >
> >
> > On 6/19/19 7:02 AM, Ron Dagostino wrote:
> > > Looks great, Colin.
> > >
> > > I have also enjoyed Stephane Maarek's "What's New in Kafka..." series
> of
> > > videos (e.g. https://www.youtube.com/watch?v=kaWbp1Cnfo4=10s).
> Having
> > > summaries like this in both formats -- blog and video -- for every
> release
> > > would be helpful as different people have different preferences.
> > >
> > > Ron
> > >
> > > On Tue, Jun 18, 2019 at 8:20 PM Colin McCabe 
> wrote:
> > >
> > >> Thanks, Konstantine.  I reworked the wording a bit -- take a look.
> > >>
> > >> best,
> > >> C.
> > >>
> > >> On Tue, Jun 18, 2019, at 14:31, Konstantine Karantasis wrote:
> > >>> Thanks Colin.
> > >>> Great initiative!
> > >>>
> > >>> Here's a small correction (between **) for KIP-415 with a small
> > >> suggestion
> > >>> as well (between _ _):
> > >>>
> > >>> In Kafka Connect, worker tasks are distributed among the available
> worker
> > >>> nodes. When a connector is reconfigured or a new connector is
> deployed
> > >> _as
> > >>> well as when a worker is added or removed_, the *tasks* must be
> > >> rebalanced
> > >>> across the Connect cluster to help ensure that all of the worker
> nodes
> > >> are
> > >>> doing a fair share of the Connect work. In 2.2 and earlier, a Connect
> > >>> rebalance caused all worker threads to pause while the rebalance
> > >> proceeded.
> > >>> As of KIP-415, rebalancing is no longer a stop-the-world affair,
> making
> > >>> configuration changes a more pleasant thing.
> > >>>
> > >>> Cheers,
> > >>> Konstantine
> > >>>
> > >>> On Tue, Jun 18, 2019 at 1:50 PM Swen Moczarski <
> swen.moczar...@gmail.com
> > >>>
> > >>> wrote:
> > >>>
> >  Nice overview!
> > 
> >  I found some typos:
> >  rbmainder
> >  bmits
> >  implbmentation
> > 
> >  Am Di., 18. Juni 2019 um 22:43 Uhr schrieb Boyang Chen <
> >  bche...@outlook.com
> > > :
> > 
> > > One typo:
> > > KIP-428: Add in-mbmory window store
> > > should be
> > > KIP-428: Add in-memory window store
> > >
> > >
> > > 
> > > From: Colin McCabe 
> > > Sent: Wednesday, June 19, 2019 4:22 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: Preliminary blog post for the Apache Kafka 2.3.0
> release
> > >
> > > Sorry, I copied the wrong URL at first.  Try this URL instead:
> > >
> > 
> > >>
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache
> > >
> > > best,
> > > Colin
> > >
> > > On Tue, Jun 18, 2019, at 13:17, Colin McCabe wrote:
> > >> Hmm.  I'm looking to see if there's any way to open up the
> > > permissions... :|
> > >>
> > >>
> > >>
> > >> On Tue, Jun 18, 2019, at 13:12, M. Manna wrote:
> > >>> It’s asking for credentials...?
> > >>>
> > >>> On Tue, 18 Jun 2019 at 15:10, Colin McCabe 
> >  wrote:
> > >>>
> >  Hi all,
> > 
> >  I've written up a preliminary blog post about the upcoming
> > >> Apache
> > > Kafka
> >  2.3.0 release.  Take a look and let me know what you think.
> > 
> > 
> > 
> > >
> > 
> > >>
> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache
> > 
> >  cheers,
> >  Colin
> > 
> > >>>
> > >>
> > >
> > 
> > >>>
> > >>
> > >
> >
> >
> > Attachments:
> > * signature.asc
>

[jira] [Created] (KAFKA-8586) Source task producers silently fail to send records

[Chris Egerton (JIRA)]

Chris Egerton created KAFKA-8586:


 Summary: Source task producers silently fail to send records
 Key: KAFKA-8586
 URL: https://issues.apache.org/jira/browse/KAFKA-8586
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 2.3.0
Reporter: Chris Egerton


The Connect framework marks source records as successfully sent when they are 
dispatched to the producer, instead of when they are actually sent to Kafka. 
[This is assumed to be good 
enough|https://github.com/apache/kafka/blob/3e9d1c1411c5268de382f9dfcc95bdf66d0063a0/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java#L324-L331]
 since the Connect framework sets up its producer to use infinite retries on 
retriable errors, but in the case of an authorization or authentication failure 
with a secured Kafka broker, the errors aren't retriable and cause the producer 
to invoke its send callback with an exception and then give up on sending the 
message. This is a problem since the callback currently used by the 
WorkerSourceTask class when it invokes Producer.send(...) logs the exception 
and does nothing else. This leads to data loss since the source offsets for 
those failed records are committed, and the status of the task is never 
affected so users may not even know that something is wrong unless they check 
the worker log files or notice that data isn't flowing into Kafka. Until and 
unless someone does notice that something's wrong, the task will continue 
processing records and committing offsets, even though nothing is making it 
into Kafka.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-8585) Controller should make LeaderAndIsr updates optimistically

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-8585:
--

 Summary: Controller should make LeaderAndIsr updates optimistically
 Key: KAFKA-8585
 URL: https://issues.apache.org/jira/browse/KAFKA-8585
 Project: Kafka
  Issue Type: Improvement
Reporter: Jason Gustafson
Assignee: Jason Gustafson


Before the controller updates LeaderAndIsr information for a set of partitions, 
it always first looks up the current state. We can skip this since the state is 
already cached on the controller. In the common case (e.g. controlled 
shutdown), the update will succeed. If there was a change which had not been 
propagated to the controller, we can retry (this logic exists already).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

[John Roesler]

After kicking the naming around a bit more, it seems like any package
name change is a bit "weird" because it fragments the package and
directory structure. If we can come up with a reasonable name for the
interface after all, it seems like the better choice.

The real challenge is that the existing name "Processor" seems just
about perfect. In picking a new name, we need to consider the ultimate
state, after the deprecation period, when we entirely remove
Processor. In this context, TypedProcessor seems a little odd to me,
because it seems to imply that there should also be an "untyped
processor".

After kicking around a few other ideas, what does everyone think about
"RecordProcessor"? I _think_ maybe it stands on its own just fine,
because it's a thing that processes... records?

If others agree with this, I can change the proposal to RecordProcessor.

Thanks,
-John

On Fri, Jun 21, 2019 at 6:42 PM John Roesler  wrote:
>
> Hi all,
>
> I've updated the KIP with the feedback so far.
>
> The naming question is still the biggest (only?) outstanding issue. It
> would be good to hear some more thoughts on it.
>
> As we stand now, there's one vote for changing the package name to
> something like 'typedprocessor', one for changing the interface to
> TypedProcessor (as in the PoC), and one for just changing the
> Processor interface in-place, breaking source compatibility.
>
> How can we resolve this decision?
>
> Thanks,
> -John
>
> On Thu, Jun 20, 2019 at 5:44 PM John Roesler  wrote:
> >
> > Thanks for the feedback, Guozhang and Matthias,
> >
> > Regarding motivation: I'll update the wiki. Briefly:
> > * Any processor can benefit. Imagine a pure user of the ProcessorAPI
> > who has very complex processing logic. I have seen several processor
> > implementation that are hundreds of lines long and call
> > `context.forward` in many different locations and branches. In such an
> > implementation, it would be very easy to have a bug in a rarely used
> > branch that forwards the wrong kind of value. This would structurally
> > prevent that from happening.
> > * Also, anyone who heavily uses the ProcessorAPI would likely have
> > developed helper methods to wire together processors, just as we have
> > in the DSL implementation. This change would enable them to ensure at
> > compile time that they are actually wiring together compatible types.
> > This was actually _my_ original motivation, since I found it very
> > difficult and time consuming to follow the Streams DSL internal
> > builders.
> >
> > Regarding breaking the source compatibility of Processor: I would
> > _love_ to side-step the naming problem, but I really don't know if
> > it's excusable to break compatibility. I suspect that our oldest and
> > dearest friends are using the ProcessorAPI in some form or another,
> > and all their source code would break. It sucks to have to create a
> > whole new interface to get around this, but it feels like the right
> > thing to do. Would be nice to get even more feedback on this point,
> > though.
> >
> > Regarding the types of stores, as I said in my response to Sophie,
> > it's not an issue.
> >
> > Regarding the change to StreamsBuilder, it doesn't pin the types in
> > any way, since all the types are bounded by Object only, and there are
> > no extra constraints between arguments (each type is used only once in
> > one argument). But maybe I missed the point you were asking about.
> > Since the type takes generic paramters, we should allow users to pass
> > in parameterized arguments. Otherwise, they would _have to_ give us a
> > raw type, and they would be forced to get a "rawtyes" warning from the
> > compiler. So, it's our obligation in any API that accepts a
> > parameterized-type parameter to allow people to actually pass a
> > parameterized type, even if we don't actually use the parameters.
> >
> > The naming question is a complex one, as I took pains to detail
> > previously. Please don't just pick out one minor point, call it weak,
> > and then claim that it invalidates the whole decision. I don't think
> > there's a clear best choice, so I'm more than happy for someone to
> > advocate for renaming the class instead of the package. Can you
> > provide some reasons why you think that would be better?
> >
> > Regarding the deprecated methods, you're absolutely right. I'll update the 
> > KIP.
> >
> > Thanks again for all the feedback!
> > -John
> >
> > On Thu, Jun 20, 2019 at 4:34 PM Matthias J. Sax  
> > wrote:
> > >
> > > Just want to second what Sophie said about the stores. The type of a
> > > used stores is completely independent of input/output types.
> > >
> > > This related to change `addGlobalStore()` method. Why do you want to pin
> > > the types? In fact, people request the ability to filter() and maybe
> > > even map() the data before they are put into the global store. Limiting
> > > the types seems to be a step backward here?
> > >
> > >
> > >
> > > Also, the pack name is questionable.

Re: [VOTE] 2.3.0 RC3

[Guozhang Wang]

Hi Colin,

Thanks for the new RC, +1 (binding).

Verified the javadoc, maven repo, and ran unit tests on 2.12 binary.


Guozhang


On Fri, Jun 21, 2019 at 1:23 PM Colin McCabe  wrote:

> Hi Ismael,
>
> Good catch.  This should be fixed now.
>
> It seems that if the previously staged Sonatype staging repositories (from
> the previous RCs) are not dropped as part of the release process, the new
> ones don't get exposed.  Maybe we should document this somewhere (it was a
> bit of a surprise when I first encountered it.)  Maybe the release.py
> script should mention it.
>
> best,
> Colin
>
>
> On Fri, Jun 21, 2019, at 08:30, Ismael Juma wrote:
> > Hi Colin,
> >
> > One more thing: the Maven repo was not updated, it seems. Can you fix
> that?
> >
> > Ismael
> >
> > On Fri, Jun 21, 2019 at 8:23 AM Ismael Juma  wrote:
> >
> > > Hi Colin,
> > >
> > > Thanks, +1 (binding).
> > >
> > > I verified the quickstart and signatures for the source and kafka 2.12
> > > artifacts with Java 11. One note is that we need to complete the
> upgrade
> > > section for 2.3.0 before we announce the release. No new RC is needed,
> we
> > > just have to have it on the website.
> > >
> > > Ismael
> > >
> > > On Wed, Jun 19, 2019 at 3:03 PM Colin McCabe 
> wrote:
> > >
> > >> Hi all,
> > >>
> > >> We discovered some problems with the second release candidate (RC2) of
> > >> 2.3.0.  Specifically, KAFKA-8564.  I've created a new RC which
> includes the
> > >> fix for this issue.
> > >>
> > >> Check out the release notes for the 2.3.0 release here:
> > >> https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/RELEASE_NOTES.html
> > >>
> > >> The vote will go until Saturday, June 22nd, or until we create
> another RC.
> > >>
> > >> * Kafka's KEYS file containing PGP keys we use to sign the release
> can be
> > >> found here:
> > >> https://kafka.apache.org/KEYS
> > >>
> > >> * The release artifacts to be voted upon (source and binary) are here:
> > >> https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/
> > >>
> > >> * Maven artifacts to be voted upon:
> > >>
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >>
> > >> * Javadoc:
> > >> https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/javadoc/
> > >>
> > >> * The tag to be voted upon (off the 2.3 branch) is the 2.3.0 tag:
> > >> https://github.com/apache/kafka/releases/tag/2.3.0-rc3
> > >>
> > >> best,
> > >> Colin
> > >>
> > >> C.
> > >>
> > >
> >
>


-- 
-- Guozhang

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

[John Roesler]

Hi all,

I've updated the KIP with the feedback so far.

The naming question is still the biggest (only?) outstanding issue. It
would be good to hear some more thoughts on it.

As we stand now, there's one vote for changing the package name to
something like 'typedprocessor', one for changing the interface to
TypedProcessor (as in the PoC), and one for just changing the
Processor interface in-place, breaking source compatibility.

How can we resolve this decision?

Thanks,
-John

On Thu, Jun 20, 2019 at 5:44 PM John Roesler  wrote:
>
> Thanks for the feedback, Guozhang and Matthias,
>
> Regarding motivation: I'll update the wiki. Briefly:
> * Any processor can benefit. Imagine a pure user of the ProcessorAPI
> who has very complex processing logic. I have seen several processor
> implementation that are hundreds of lines long and call
> `context.forward` in many different locations and branches. In such an
> implementation, it would be very easy to have a bug in a rarely used
> branch that forwards the wrong kind of value. This would structurally
> prevent that from happening.
> * Also, anyone who heavily uses the ProcessorAPI would likely have
> developed helper methods to wire together processors, just as we have
> in the DSL implementation. This change would enable them to ensure at
> compile time that they are actually wiring together compatible types.
> This was actually _my_ original motivation, since I found it very
> difficult and time consuming to follow the Streams DSL internal
> builders.
>
> Regarding breaking the source compatibility of Processor: I would
> _love_ to side-step the naming problem, but I really don't know if
> it's excusable to break compatibility. I suspect that our oldest and
> dearest friends are using the ProcessorAPI in some form or another,
> and all their source code would break. It sucks to have to create a
> whole new interface to get around this, but it feels like the right
> thing to do. Would be nice to get even more feedback on this point,
> though.
>
> Regarding the types of stores, as I said in my response to Sophie,
> it's not an issue.
>
> Regarding the change to StreamsBuilder, it doesn't pin the types in
> any way, since all the types are bounded by Object only, and there are
> no extra constraints between arguments (each type is used only once in
> one argument). But maybe I missed the point you were asking about.
> Since the type takes generic paramters, we should allow users to pass
> in parameterized arguments. Otherwise, they would _have to_ give us a
> raw type, and they would be forced to get a "rawtyes" warning from the
> compiler. So, it's our obligation in any API that accepts a
> parameterized-type parameter to allow people to actually pass a
> parameterized type, even if we don't actually use the parameters.
>
> The naming question is a complex one, as I took pains to detail
> previously. Please don't just pick out one minor point, call it weak,
> and then claim that it invalidates the whole decision. I don't think
> there's a clear best choice, so I'm more than happy for someone to
> advocate for renaming the class instead of the package. Can you
> provide some reasons why you think that would be better?
>
> Regarding the deprecated methods, you're absolutely right. I'll update the 
> KIP.
>
> Thanks again for all the feedback!
> -John
>
> On Thu, Jun 20, 2019 at 4:34 PM Matthias J. Sax  wrote:
> >
> > Just want to second what Sophie said about the stores. The type of a
> > used stores is completely independent of input/output types.
> >
> > This related to change `addGlobalStore()` method. Why do you want to pin
> > the types? In fact, people request the ability to filter() and maybe
> > even map() the data before they are put into the global store. Limiting
> > the types seems to be a step backward here?
> >
> >
> >
> > Also, the pack name is questionable.
> >
> > > This wouldn't be the first project to do something like this...
> >
> > Not a strong argument. I would actually propose to not a a new package,
> > but just a new class `TypedProcessor`.
> >
> >
> > For `ProcessorContext#forward` methods -- some of those methods are
> > already deprecated. While the will still be affected, it would be worth
> > to mark them as deprecated in the wiki page, too.
> >
> >
> > @Guozhang: I dont' think we should break source compatibility in a minor
> > release.
> >
> >
> >
> > -Matthias
> >
> >
> >
> > On 6/20/19 1:43 PM, Guozhang Wang wrote:
> > > Hi John,
> > >
> > > Thanks for KIP! I've a few comments below:
> > >
> > > 1. So far the "Motivation" section is very general, and the only concrete
> > > example that I have in mind is `TransformValues#punctuate`. Do we have any
> > > other concrete issues that drive this KIP? If not then I feel better to
> > > narrow the scope of this KIP to:
> > >
> > > 1.a) modifying ProcessorContext only with the output types on forward.
> > > 1.b) modifying Transformer signature to have generics of ProcessorContext,
> > 

[jira] [Created] (KAFKA-8584) Allow "bytes" type to generated a ByteBuffer rather than byte arrays

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-8584:


 Summary: Allow "bytes" type to generated a ByteBuffer rather than 
byte arrays
 Key: KAFKA-8584
 URL: https://issues.apache.org/jira/browse/KAFKA-8584
 Project: Kafka
  Issue Type: Sub-task
Reporter: Guozhang Wang


Right now in the RPC definition, type {{bytes}} would be translated into 
{{byte[]}} in generated Java code. However, for some requests like 
ProduceRequest#partitionData, the underlying type would better be a ByteBuffer 
rather than a byte array.

One proposal is to add an additional boolean tag {{useByteBuffer}} for 
{{bytes}} type, which by default is false; when set to {{true}} set the 
corresponding field to generate {{ByteBuffer}} instead of {{[]byte}}. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-476: Add Java AdminClient interface

[Matthias J. Sax]

I still think, that an interface does not need to know anything about
its implementation. But I am also fine if we add a factory method to the
new interface if that is preferred by most people.


-Matthias

On 6/21/19 7:10 AM, Ismael Juma wrote:
> This is even more reason not to deprecate immediately, there is very little
> maintenance cost for us. We should be mindful that many of our users (eg
> Spark, Flink, etc.) typically allow users to specify the kafka clients
> version and hence avoid using new classes/interfaces for some time. They
> would get a bunch of warnings they cannot do anything about apart from
> suppressing.
> 
> Ismael
> 
> On Fri, Jun 21, 2019 at 4:00 AM Andy Coates  wrote:
> 
>> Hi Ismael,
>>
>> I’m happy enough to not deprecate the existing `AdminClient` class as part
>> of this change.
>>
>> However, note that, the class will likely be empty, i.e. all methods and
>> implementations will be inherited from the interface:
>>
>> public abstract class AdminClient implements Admin {
>> }
>>
>> Not marking it as deprecated has the benefit that users won’t see any
>> deprecation warnings on the next release. Conversely, deprecating it will
>> mean we can choose to remove this, now pointless class, in the future if we
>> choose.
>>
>> That’s my thinking for deprecation, but as I’ve said I’m happy either way.
>>
>> Andy
>>
>>> On 18 Jun 2019, at 16:09, Ismael Juma  wrote:
>>>
>>> I agree with Ryanne, I think we should avoid deprecating AdminClient and
>>> causing so much churn for users who don't actually care about this niche
>>> use case.
>>>
>>> Ismael
>>>
>>> On Tue, Jun 18, 2019 at 6:43 AM Andy Coates  wrote:
>>>
 Hi Ryanne,

 If we don't change the client code, then everywhere will still expect
 subclasses of `AdminClient`, so the interface will be of no use, i.e. I
 can't write a class that implements the new interface and pass it to the
 client code.

 Thanks,

 Andy

 On Mon, 17 Jun 2019 at 19:01, Ryanne Dolan 
>> wrote:

> Andy, while I agree that the new interface is useful, I'm not convinced
> adding an interface requires deprecating AdminClient and changing so
>> much
> client code. Why not just add the Admin interface, have AdminClient
> implement it, and have done?
>
> Ryanne
>
> On Mon, Jun 17, 2019 at 12:09 PM Andy Coates 
>> wrote:
>
>> Hi all,
>>
>> I think I've addressed all concerns. Let me know if I've not.  Can I
 call
>> another round of votes please?
>>
>> Thanks,
>>
>> Andy
>>
>> On Fri, 14 Jun 2019 at 04:55, Satish Duggana <
>> satish.dugg...@gmail.com
>
>> wrote:
>>
>>> Hi Andy,
>>> Thanks for the KIP. This is a good change and it gives the user a
> better
>>> handle on Admin client usage. I agree with the proposal except the
 new
>>> `Admin` interface having all the methods from `AdminClient` abstract
>> class.
>>> It should be kept clean having only the admin operations as methods
> from
>>> KafkaClient abstract class but not the factory methods as mentioned
 in
>> the
>>> earlier mail.
>>>
>>> I know about dynamic proxies(which were widely used in RMI/EJB
 world).
> I
>> am
>>> curious about the usecase using dynamic proxies with Admin client
>>> interface. Dynamic proxy can have performance penalty if it is used
 in
>>> critical path. Is that the primary motivation for creating the KIP?
>>>
>>> Thanks,
>>> Satish.
>>>
>>> On Wed, Jun 12, 2019 at 8:43 PM Andy Coates 
 wrote:
>>>
 I'm not married to that part.  That was only done to keep it more
 or
>> less
 inline with what's already there, (an abstract class that has a
> factory
 method that returns a subclass sounds like the same
 anti-pattern
>> ;))

 An alternative would to have an `AdminClients` utility class to
> create
>>> the
 admin client.

 On Mon, 10 Jun 2019 at 19:31, Matthias J. Sax <
 matth...@confluent.io
>>
 wrote:

> Hmmm...
>
> So the new interface, returns an instance of a class that
> implements
>>> the
> interface. This sounds a little bit like an anti-pattern?
 Shouldn't
> interfaces actually not know anything about classes that
 implement
>> the
> interface?
>
>
> -Matthias
>
> On 6/10/19 11:22 AM, Andy Coates wrote:
>> `AdminClient` would be deprecated purely because it would no
> longer
 serve
>> any purpose and would be virtually empty, getting all of its
> implementation
>> from the new interfar. It would be nice to remove this from the
> API
>>> at
> the
>> next major version bump, hence the need to deprecate.
>>
>> 

Re: [DISCUSS] KIP-470: TopologyTestDriver test input and output usability improvements

[Jukka Karvanen]

Hi Bill,

These topic objects are only interfacing TopologyTestDriver, not affecting
the internal functionality of it. In my plan the internal data structures
are using those Producer/ConsumerRecords as earlier. That way I don't see
how those could be affected.

Jukka



On Fri, 21 Jun 2019, 20:57 Bill Bejeck,  wrote:

> Jukka,
>
> Thanks for the KIP. I like the changes overall.
> One thing I wanted to confirm, and this may be me being paranoid, but will
> the changes for input/output topic affect how the TopologyTestDriver works
> with internal topics when there are sub-topologies created?
>
> On Fri, Jun 21, 2019 at 12:05 PM Guozhang Wang  wrote:
>
> > 1) Got it, could you list this class along with all its functions in the
> > proposed public APIs as well?
> >
> > 2) Ack, thanks!
> >
> > On Thu, Jun 20, 2019 at 11:27 PM Jukka Karvanen <
> > jukka.karva...@jukinimi.com>
> > wrote:
> >
> > > Hi  Guozhang,
> > >
> > > 1) This TestRecord is new class in my proposal. So it is a simplified
> > > version of ProducerRecord and ConsumerRecord containing only the fields
> > > needed to test record content.
> > >
> > > 2)
> > > public final  TestInputTopic createInputTopic(final String
> > > topicName, final Serde keySerde, final Serde valueSerde);
> > > public final  TestOutputTopic createOutputTopic(final
> String
> > > topicName, final Serde keySerde, final Serde valueSerde);
> > > The purpose is to create separate object for each input and output
> topic
> > > you are using. The topic name is given to createInput/OutputTopic when
> > > initialize topic object.
> > >
> > > For example:
> > >
> > > final TestInputTopic inputTopic1 =
> > > testDriver.createInputTopic( INPUT_TOPIC, longSerde, stringSerde);
> > > final TestInputTopic inputTopic2 =
> > > testDriver.createInputTopic( INPUT_TOPIC_MAP, longSerde, stringSerde);
> > > final TestOutputTopic outputTopic1 =
> > > testDriver.createOutputTopic(OUTPUT_TOPIC, longSerde, stringSerde);
> > > final TestOutputTopic outputTopic2 =
> > > testDriver.createOutputTopic(OUTPUT_TOPIC_MAP, stringSerde,
> > > longSerde);
> > > inputTopic1.pipeInput(1L, "Hello");
> > > assertThat(outputTopic1.readKeyValue(), equalTo(new KeyValue<>(1L,
> > > "Hello")));
> > > assertThat(outputTopic2.readKeyValue(), equalTo(new KeyValue<>("Hello",
> > > 1L)));
> > > inputTopic2.pipeInput(1L, "Hello");
> > >
> > >
> > > Jukka
> > >
> > > to 20. kesäk. 2019 klo 23.52 Guozhang Wang (wangg...@gmail.com)
> > kirjoitti:
> > >
> > > > Hello Jukka,
> > > >
> > > > Thanks for writing the KIP, I have a couple of quick questions:
> > > >
> > > > 1) Is "TestRecord" an existing class that you propose to piggy-back
> on?
> > > > Right now we have a scala TestRecord case class but I doubt that was
> > your
> > > > proposal, or are you proposing to add a new Java class?
> > > >
> > > > 2) Would the new API only allow a single input / output topic with
> > > > `createInput/OutputTopic`? If not, when we call pipeInput how to
> > > determine
> > > > which topic this record should be pipe to?
> > > >
> > > >
> > > > Guozhang
> > > >
> > > > On Mon, Jun 17, 2019 at 1:34 PM John Roesler 
> > wrote:
> > > >
> > > > > Woah, I wasn't aware of that Hamcrest test style. Awesome!
> > > > >
> > > > > Thanks for the updates. I look forward to hearing what others
> think.
> > > > >
> > > > > -John
> > > > >
> > > > > On Mon, Jun 17, 2019 at 4:12 AM Jukka Karvanen
> > > > >  wrote:
> > > > > >
> > > > > > Wiki page updated:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-470%3A+TopologyTestDriver+test+input+and+output+usability+improvements
> > > > > >
> > > > > >
> > > > > > ClientRecord removed and replaced with TestRecord in method
> calls.
> > > > > > TestRecordFactory removed (time tracking functionality to be
> > included
> > > > to
> > > > > > TestInputTopic)
> > > > > > OutputVerifier deprecated
> > > > > > TestRecord topic removed and getters added
> > > > > >
> > > > > > Getters in TestRecord enable writing test ignoring selected
> fields
> > > with
> > > > > > hamcrest like this:
> > > > > >
> > > > > > assertThat(outputTopic.readRecord(), allOf(
> > > > > > hasProperty("key", equalTo(1L)),
> > > > > > hasProperty("value", equalTo("Hello")),
> > > > > > hasProperty("headers", equalTo(headers;
> > > > > >
> > > > > >
> > > > > > Jukka
> > > > > >
> > > > > > la 15. kesäk. 2019 klo 1.10 John Roesler (j...@confluent.io)
> > > > kirjoitti:
> > > > > >
> > > > > > > Sounds good. Thanks as always for considering my feedback!
> > > > > > > -John
> > > > > > >
> > > > > > > On Fri, Jun 14, 2019 at 12:12 PM Jukka Karvanen
> > > > > > >  wrote:
> > > > > > > >
> > > > > > > > Ok, I will modify KIP Public Interface in a wiki based on the
> > > > > feedback.
> > > > > > > >
> > > > > > > > TestRecordFactory / ConsumerRecordFactory was used by
> > > > TestInputTopic
> > > > > with
> > > > > > > > the version I had with KIP456, but maybe I can merge 

Re: [DISCUSS] KIP-470: TopologyTestDriver test input and output usability improvements

[Jukka Karvanen]

Hi,

TestRecord is already included in wiki page. It has mainly constructors and
methods to access data fields.

Jukka

On Fri, 21 Jun 2019, 19:05 Guozhang Wang,  wrote:

> 1) Got it, could you list this class along with all its functions in the
> proposed public APIs as well?
>
> 2) Ack, thanks!
>
> On Thu, Jun 20, 2019 at 11:27 PM Jukka Karvanen <
> jukka.karva...@jukinimi.com>
> wrote:
>
> > Hi  Guozhang,
> >
> > 1) This TestRecord is new class in my proposal. So it is a simplified
> > version of ProducerRecord and ConsumerRecord containing only the fields
> > needed to test record content.
> >
> > 2)
> > public final  TestInputTopic createInputTopic(final String
> > topicName, final Serde keySerde, final Serde valueSerde);
> > public final  TestOutputTopic createOutputTopic(final String
> > topicName, final Serde keySerde, final Serde valueSerde);
> > The purpose is to create separate object for each input and output topic
> > you are using. The topic name is given to createInput/OutputTopic when
> > initialize topic object.
> >
> > For example:
> >
> > final TestInputTopic inputTopic1 =
> > testDriver.createInputTopic( INPUT_TOPIC, longSerde, stringSerde);
> > final TestInputTopic inputTopic2 =
> > testDriver.createInputTopic( INPUT_TOPIC_MAP, longSerde, stringSerde);
> > final TestOutputTopic outputTopic1 =
> > testDriver.createOutputTopic(OUTPUT_TOPIC, longSerde, stringSerde);
> > final TestOutputTopic outputTopic2 =
> > testDriver.createOutputTopic(OUTPUT_TOPIC_MAP, stringSerde,
> > longSerde);
> > inputTopic1.pipeInput(1L, "Hello");
> > assertThat(outputTopic1.readKeyValue(), equalTo(new KeyValue<>(1L,
> > "Hello")));
> > assertThat(outputTopic2.readKeyValue(), equalTo(new KeyValue<>("Hello",
> > 1L)));
> > inputTopic2.pipeInput(1L, "Hello");
> >
> >
> > Jukka
> >
> > to 20. kesäk. 2019 klo 23.52 Guozhang Wang (wangg...@gmail.com)
> kirjoitti:
> >
> > > Hello Jukka,
> > >
> > > Thanks for writing the KIP, I have a couple of quick questions:
> > >
> > > 1) Is "TestRecord" an existing class that you propose to piggy-back on?
> > > Right now we have a scala TestRecord case class but I doubt that was
> your
> > > proposal, or are you proposing to add a new Java class?
> > >
> > > 2) Would the new API only allow a single input / output topic with
> > > `createInput/OutputTopic`? If not, when we call pipeInput how to
> > determine
> > > which topic this record should be pipe to?
> > >
> > >
> > > Guozhang
> > >
> > > On Mon, Jun 17, 2019 at 1:34 PM John Roesler 
> wrote:
> > >
> > > > Woah, I wasn't aware of that Hamcrest test style. Awesome!
> > > >
> > > > Thanks for the updates. I look forward to hearing what others think.
> > > >
> > > > -John
> > > >
> > > > On Mon, Jun 17, 2019 at 4:12 AM Jukka Karvanen
> > > >  wrote:
> > > > >
> > > > > Wiki page updated:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-470%3A+TopologyTestDriver+test+input+and+output+usability+improvements
> > > > >
> > > > >
> > > > > ClientRecord removed and replaced with TestRecord in method calls.
> > > > > TestRecordFactory removed (time tracking functionality to be
> included
> > > to
> > > > > TestInputTopic)
> > > > > OutputVerifier deprecated
> > > > > TestRecord topic removed and getters added
> > > > >
> > > > > Getters in TestRecord enable writing test ignoring selected fields
> > with
> > > > > hamcrest like this:
> > > > >
> > > > > assertThat(outputTopic.readRecord(), allOf(
> > > > > hasProperty("key", equalTo(1L)),
> > > > > hasProperty("value", equalTo("Hello")),
> > > > > hasProperty("headers", equalTo(headers;
> > > > >
> > > > >
> > > > > Jukka
> > > > >
> > > > > la 15. kesäk. 2019 klo 1.10 John Roesler (j...@confluent.io)
> > > kirjoitti:
> > > > >
> > > > > > Sounds good. Thanks as always for considering my feedback!
> > > > > > -John
> > > > > >
> > > > > > On Fri, Jun 14, 2019 at 12:12 PM Jukka Karvanen
> > > > > >  wrote:
> > > > > > >
> > > > > > > Ok, I will modify KIP Public Interface in a wiki based on the
> > > > feedback.
> > > > > > >
> > > > > > > TestRecordFactory / ConsumerRecordFactory was used by
> > > TestInputTopic
> > > > with
> > > > > > > the version I had with KIP456, but maybe I can merge That
> > > > functionality
> > > > > > to
> > > > > > > InputTopic or  TestRecordFactory   can kept non public maybe
> > moving
> > > > it to
> > > > > > > internals package.
> > > > > > >
> > > > > > > I will make the proposal with a slim down interface.
> > > > > > > I don't want to go to so slim as you proposed with only
> > TestRecord
> > > or
> > > > > > > List, because you then still end up doing helper
> > > methods
> > > > to
> > > > > > > construct List of TestRecord.
> > > > > > > The list of values is easier to write and clearer to read than
> if
> > > you
> > > > > > need
> > > > > > > to contruct list of TestRecords.
> > > > > > >
> > > > > > > For example:
> > > > > > >
> > > > > > > final List inputValues = Arrays.asList(

Jenkins build is back to normal : kafka-trunk-jdk11 #650

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-8583) Optimization for SslTransportLayer#write(ByteBuffer)

[Mao (JIRA)]

Mao created KAFKA-8583:
--

 Summary: Optimization for SslTransportLayer#write(ByteBuffer)
 Key: KAFKA-8583
 URL: https://issues.apache.org/jira/browse/KAFKA-8583
 Project: Kafka
  Issue Type: Improvement
  Components: network
Reporter: Mao


The is a propose to optimize SslTransportLayer#write(ByteBuffer).

Currently, sslEngine only reads data from src once per selector loop.

This can be optimized by reading all src data in one selector loop.

The propose comes from [Amrby|[https://github.com/linkedin/ambry/pull/1105]], 
which uses same Transport Layer code. 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-2.3-jdk8 #55

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-8570; Grow buffer to hold down converted records if it was

--
[...truncated 2.94 MB...]
kafka.zk.KafkaZkClientTest > testLogDirGetters PASSED

kafka.zk.KafkaZkClientTest > testSetGetAndDeletePartitionReassignment STARTED

kafka.zk.KafkaZkClientTest > testSetGetAndDeletePartitionReassignment PASSED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationsDeletion STARTED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationsDeletion PASSED

kafka.zk.KafkaZkClientTest > testGetDataAndVersion STARTED

kafka.zk.KafkaZkClientTest > testGetDataAndVersion PASSED

kafka.zk.KafkaZkClientTest > testGetChildren STARTED

kafka.zk.KafkaZkClientTest > testGetChildren PASSED

kafka.zk.KafkaZkClientTest > testSetAndGetConsumerOffset STARTED

kafka.zk.KafkaZkClientTest > testSetAndGetConsumerOffset PASSED

kafka.zk.KafkaZkClientTest > testClusterIdMethods STARTED

kafka.zk.KafkaZkClientTest > testClusterIdMethods PASSED

kafka.zk.KafkaZkClientTest > testEntityConfigManagementMethods STARTED

kafka.zk.KafkaZkClientTest > testEntityConfigManagementMethods PASSED

kafka.zk.KafkaZkClientTest > testUpdateLeaderAndIsr STARTED

kafka.zk.KafkaZkClientTest > testUpdateLeaderAndIsr PASSED

kafka.zk.KafkaZkClientTest > testUpdateBrokerInfo STARTED

kafka.zk.KafkaZkClientTest > testUpdateBrokerInfo PASSED

kafka.zk.KafkaZkClientTest > testCreateRecursive STARTED

kafka.zk.KafkaZkClientTest > testCreateRecursive PASSED

kafka.zk.KafkaZkClientTest > testGetConsumerOffsetNoData STARTED

kafka.zk.KafkaZkClientTest > testGetConsumerOffsetNoData PASSED

kafka.zk.KafkaZkClientTest > testDeleteTopicPathMethods STARTED

kafka.zk.KafkaZkClientTest > testDeleteTopicPathMethods PASSED

kafka.zk.KafkaZkClientTest > testSetTopicPartitionStatesRaw STARTED

kafka.zk.KafkaZkClientTest > testSetTopicPartitionStatesRaw PASSED

kafka.zk.KafkaZkClientTest > testAclManagementMethods STARTED

kafka.zk.KafkaZkClientTest > testAclManagementMethods PASSED

kafka.zk.KafkaZkClientTest > testPreferredReplicaElectionMethods STARTED

kafka.zk.KafkaZkClientTest > testPreferredReplicaElectionMethods PASSED

kafka.zk.KafkaZkClientTest > testPropagateLogDir STARTED

kafka.zk.KafkaZkClientTest > testPropagateLogDir PASSED

kafka.zk.KafkaZkClientTest > testGetDataAndStat STARTED

kafka.zk.KafkaZkClientTest > testGetDataAndStat PASSED

kafka.zk.KafkaZkClientTest > testReassignPartitionsInProgress STARTED

kafka.zk.KafkaZkClientTest > testReassignPartitionsInProgress PASSED

kafka.zk.KafkaZkClientTest > testCreateTopLevelPaths STARTED

kafka.zk.KafkaZkClientTest > testCreateTopLevelPaths PASSED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationGetters STARTED

kafka.zk.KafkaZkClientTest > testIsrChangeNotificationGetters PASSED

kafka.zk.KafkaZkClientTest > testLogDirEventNotificationsDeletion STARTED

kafka.zk.KafkaZkClientTest > testLogDirEventNotificationsDeletion PASSED

kafka.zk.KafkaZkClientTest > testGetLogConfigs STARTED

kafka.zk.KafkaZkClientTest > testGetLogConfigs PASSED

kafka.zk.KafkaZkClientTest > testBrokerSequenceIdMethods STARTED

kafka.zk.KafkaZkClientTest > testBrokerSequenceIdMethods PASSED

kafka.zk.KafkaZkClientTest > testAclMethods STARTED

kafka.zk.KafkaZkClientTest > testAclMethods PASSED

kafka.zk.KafkaZkClientTest > testCreateSequentialPersistentPath STARTED

kafka.zk.KafkaZkClientTest > testCreateSequentialPersistentPath PASSED

kafka.zk.KafkaZkClientTest > testConditionalUpdatePath STARTED

kafka.zk.KafkaZkClientTest > testConditionalUpdatePath PASSED

kafka.zk.KafkaZkClientTest > testDeleteTopicZNode STARTED

kafka.zk.KafkaZkClientTest > testDeleteTopicZNode PASSED

kafka.zk.KafkaZkClientTest > testDeletePath STARTED

kafka.zk.KafkaZkClientTest > testDeletePath PASSED

kafka.zk.KafkaZkClientTest > testGetBrokerMethods STARTED

kafka.zk.KafkaZkClientTest > testGetBrokerMethods PASSED

kafka.zk.KafkaZkClientTest > testCreateTokenChangeNotification STARTED

kafka.zk.KafkaZkClientTest > testCreateTokenChangeNotification PASSED

kafka.zk.KafkaZkClientTest > testGetTopicsAndPartitions STARTED

kafka.zk.KafkaZkClientTest > testGetTopicsAndPartitions PASSED

kafka.zk.KafkaZkClientTest > testRegisterBrokerInfo STARTED

kafka.zk.KafkaZkClientTest > testRegisterBrokerInfo PASSED

kafka.zk.KafkaZkClientTest > testRetryRegisterBrokerInfo STARTED

kafka.zk.KafkaZkClientTest > testRetryRegisterBrokerInfo PASSED

kafka.zk.KafkaZkClientTest > testConsumerOffsetPath STARTED

kafka.zk.KafkaZkClientTest > testConsumerOffsetPath PASSED

kafka.zk.KafkaZkClientTest > testDeleteRecursiveWithControllerEpochVersionCheck 
STARTED

kafka.zk.KafkaZkClientTest > testDeleteRecursiveWithControllerEpochVersionCheck 
PASSED

kafka.zk.KafkaZkClientTest > testControllerManagementMethods STARTED

kafka.zk.KafkaZkClientTest > testControllerManagementMethods PASSED

kafka.zk.KafkaZkClientTest > testTopicAssignmentMethods 

Build failed in Jenkins: kafka-trunk-jdk8 #3742

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-8570; Grow buffer to hold down converted records if it was

[rajinisivaram] KAFKA-8557: system tests - add support for (optional) 
interbroker

[ismael] KAFKA-8563: Remove redundant `NetworkSend.sizeDelimit()` method (#6967)

--
[...truncated 2.52 MB...]

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a store name should create a Materialized with Serdes and a store name 
STARTED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a store name should create a Materialized with Serdes and a store name 
PASSED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a window store supplier should create a Materialized with Serdes and a 
store supplier STARTED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a window store supplier should create a Materialized with Serdes and a 
store supplier PASSED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a key value store supplier should create a Materialized with Serdes and a 
store supplier STARTED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a key value store supplier should create a Materialized with Serdes and a 
store supplier PASSED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a session store supplier should create a Materialized with Serdes and a 
store supplier STARTED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a session store supplier should create a Materialized with Serdes and a 
store supplier PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > filter a KTable should 
filter records satisfying the predicate STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > filter a KTable should 
filter records satisfying the predicate PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > filterNot a KTable should 
filter records not satisfying the predicate STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > filterNot a KTable should 
filter records not satisfying the predicate PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables should join 
correctly records STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables should join 
correctly records PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables with a 
Materialized should join correctly records and state store STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables with a 
Materialized should join correctly records and state store PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilTimeLimit STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilTimeLimit PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilWindowCloses STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilWindowCloses PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > session windowed 
KTable#suppress should correctly suppress results using 
Suppressed.untilWindowCloses STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > session windowed 
KTable#suppress should correctly suppress results using 
Suppressed.untilWindowCloses PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > non-windowed 
KTable#suppress should correctly suppress results using 
Suppressed.untilTimeLimit STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > non-windowed 
KTable#suppress should correctly suppress results using 
Suppressed.untilTimeLimit PASSED

org.apache.kafka.streams.scala.kstream.SuppressedTest > 
Suppressed.untilWindowCloses should produce the correct suppression STARTED

org.apache.kafka.streams.scala.kstream.SuppressedTest > 
Suppressed.untilWindowCloses should produce the correct suppression PASSED

org.apache.kafka.streams.scala.kstream.SuppressedTest > 
Suppressed.untilTimeLimit should produce the correct suppression STARTED

org.apache.kafka.streams.scala.kstream.SuppressedTest > 
Suppressed.untilTimeLimit should produce the correct suppression PASSED

org.apache.kafka.streams.scala.kstream.SuppressedTest > BufferConfig.maxRecords 
should produce the correct buffer config STARTED

org.apache.kafka.streams.scala.kstream.SuppressedTest > BufferConfig.maxRecords 
should produce the correct buffer config PASSED

org.apache.kafka.streams.scala.kstream.SuppressedTest > BufferConfig.maxBytes 
should produce the correct 

Re: Possible implementation for KAFKA-560

[Colin McCabe]

I don't think this requires a change in the protocol.  It seems like you should 
be able to use the high water mark to figure something out here?

best,
Colin


On Fri, Jun 21, 2019, at 04:56, Carlos Manuel Duclos-Vergara wrote:
> Hi,
> 
> This is an ancient task, but I feel it is still current today (specially
> since as somebody that deals with a Kafka cluster I know that this happens
> more often than not).
> 
> The task is about garbage collection of topics in a sort of automated way.
> After some consideration I started a prototype implementation based on a
> manual process:
> 
> 1. Using the cli, I can use the --describe-topic to get a list of topics
> that have size 0
> 2. Massage that list into something that can be then fed into the cli and
> remove the topics that have size 0.
> 
> The guiding principle here is the assumption that abandoned topics will
> eventually have size 0, because all records will expire. This is not true
> for all topics, but it covers a large portion of them and having something
> like this would help admins to find "suspicious" topics at least.
> 
> I started implementing this change and I realized that it would require a
> change in the protocol, because the sizes are never sent over the wire.
> Funny enough we collect the sizes of the log files, but we do not send them.
> 
> I think this kind of changes will require a KIP, but I wanted to ask what
> others think about this.
> 
> The in-progress implementation of this can be found here:
> https://github.com/carlosduclos/kafka/commit/0dffe5e131c3bd32b77f56b9be8eded89a96df54
> 
> Comments?
> 
> -- 
> Carlos Manuel Duclos Vergara
> Backend Software Developer
>

Build failed in Jenkins: kafka-2.1-jdk8 #207

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-8570; Grow buffer to hold down converted records if it was

--
[...truncated 927.02 KB...]

kafka.server.KafkaConfigTest > testListenerAndAdvertisedListenerNames STARTED

kafka.server.KafkaConfigTest > testListenerAndAdvertisedListenerNames PASSED

kafka.server.KafkaConfigTest > testNonroutableAdvertisedListeners STARTED

kafka.server.KafkaConfigTest > testNonroutableAdvertisedListeners PASSED

kafka.server.KafkaConfigTest > 
testInterBrokerListenerNameAndSecurityProtocolSet STARTED

kafka.server.KafkaConfigTest > 
testInterBrokerListenerNameAndSecurityProtocolSet PASSED

kafka.server.KafkaConfigTest > testFromPropsInvalid STARTED

kafka.server.KafkaConfigTest > testFromPropsInvalid PASSED

kafka.server.KafkaConfigTest > testInvalidCompressionType STARTED

kafka.server.KafkaConfigTest > testInvalidCompressionType PASSED

kafka.server.KafkaConfigTest > testAdvertiseHostNameDefault STARTED

kafka.server.KafkaConfigTest > testAdvertiseHostNameDefault PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeMinutesProvided STARTED

kafka.server.KafkaConfigTest > testLogRetentionTimeMinutesProvided PASSED

kafka.server.KafkaConfigTest > testValidCompressionType STARTED

kafka.server.KafkaConfigTest > testValidCompressionType PASSED

kafka.server.KafkaConfigTest > testUncleanElectionInvalid STARTED

kafka.server.KafkaConfigTest > testUncleanElectionInvalid PASSED

kafka.server.KafkaConfigTest > testListenerNamesWithAdvertisedListenerUnset 
STARTED

kafka.server.KafkaConfigTest > testListenerNamesWithAdvertisedListenerUnset 
PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndMsProvided 
STARTED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndMsProvided 
PASSED

kafka.server.KafkaConfigTest > testLogRollTimeMsProvided STARTED

kafka.server.KafkaConfigTest > testLogRollTimeMsProvided PASSED

kafka.server.KafkaConfigTest > testUncleanLeaderElectionDefault STARTED

kafka.server.KafkaConfigTest > testUncleanLeaderElectionDefault PASSED

kafka.server.KafkaConfigTest > testInvalidAdvertisedListenersProtocol STARTED

kafka.server.KafkaConfigTest > testInvalidAdvertisedListenersProtocol PASSED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled STARTED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled PASSED

kafka.server.KafkaConfigTest > testInterBrokerVersionMessageFormatCompatibility 
STARTED

kafka.server.KafkaConfigTest > testInterBrokerVersionMessageFormatCompatibility 
PASSED

kafka.server.KafkaConfigTest > testAdvertisePortDefault STARTED

kafka.server.KafkaConfigTest > testAdvertisePortDefault PASSED

kafka.server.KafkaConfigTest > testVersionConfiguration STARTED

kafka.server.KafkaConfigTest > testVersionConfiguration PASSED

kafka.server.KafkaConfigTest > testEqualAdvertisedListenersProtocol STARTED

kafka.server.KafkaConfigTest > testEqualAdvertisedListenersProtocol PASSED

kafka.server.ListOffsetsRequestTest > testListOffsetsErrorCodes STARTED

kafka.server.ListOffsetsRequestTest > testListOffsetsErrorCodes PASSED

kafka.server.ListOffsetsRequestTest > testCurrentEpochValidation STARTED

kafka.server.ListOffsetsRequestTest > testCurrentEpochValidation PASSED

kafka.server.CreateTopicsRequestTest > testValidCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testValidCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testErrorCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testErrorCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testInvalidCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testInvalidCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testNotController STARTED

kafka.server.CreateTopicsRequestTest > testNotController PASSED

kafka.server.CreateTopicsRequestWithPolicyTest > testValidCreateTopicsRequests 
STARTED

kafka.server.CreateTopicsRequestWithPolicyTest > testValidCreateTopicsRequests 
PASSED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
STARTED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
PASSED

kafka.server.FetchRequestDownConversionConfigTest > 
testV1FetchWithDownConversionDisabled STARTED

kafka.server.FetchRequestDownConversionConfigTest > 
testV1FetchWithDownConversionDisabled PASSED

kafka.server.FetchRequestDownConversionConfigTest > testV1FetchFromReplica 
STARTED

kafka.server.FetchRequestDownConversionConfigTest > testV1FetchFromReplica 
PASSED

kafka.server.FetchRequestDownConversionConfigTest > 
testLatestFetchWithDownConversionDisabled STARTED

kafka.server.FetchRequestDownConversionConfigTest > 
testLatestFetchWithDownConversionDisabled PASSED

kafka.server.FetchRequestDownConversionConfigTest > 
testV1FetchWithTopicLevelOverrides STARTED

Build failed in Jenkins: kafka-2.2-jdk8 #141

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-8570; Grow buffer to hold down converted records if it was

--
[...truncated 2.75 MB...]
kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testReassignPartitionLeaderElectionWithEmptyIsr PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testControlledShutdownPartitionLeaderElectionAllIsrSimultaneouslyShutdown 
STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testControlledShutdownPartitionLeaderElectionAllIsrSimultaneouslyShutdown PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testOfflinePartitionLeaderElectionLastIsrOfflineUncleanLeaderElectionEnabled 
STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testOfflinePartitionLeaderElectionLastIsrOfflineUncleanLeaderElectionEnabled 
PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testPreferredReplicaPartitionLeaderElectionPreferredReplicaNotInIsrNotLive 
STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testPreferredReplicaPartitionLeaderElectionPreferredReplicaNotInIsrNotLive 
PASSED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testOfflinePartitionLeaderElectionLastIsrOfflineUncleanLeaderElectionDisabled 
STARTED

kafka.controller.PartitionLeaderElectionAlgorithmsTest > 
testOfflinePartitionLeaderElectionLastIsrOfflineUncleanLeaderElectionDisabled 
PASSED

kafka.controller.PartitionStateMachineTest > 
testNonexistentPartitionToNewPartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testNonexistentPartitionToNewPartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransitionErrorCodeFromCreateStates STARTED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransitionErrorCodeFromCreateStates PASSED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToNonexistentPartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToNonexistentPartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOfflineTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOfflineTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOfflinePartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOfflinePartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > testUpdatingOfflinePartitionsCount 
STARTED

kafka.controller.PartitionStateMachineTest > testUpdatingOfflinePartitionsCount 
PASSED

kafka.controller.PartitionStateMachineTest > 
testInvalidNonexistentPartitionToOnlinePartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testInvalidNonexistentPartitionToOnlinePartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testInvalidNonexistentPartitionToOfflinePartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testInvalidNonexistentPartitionToOfflinePartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOnlineTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOnlineTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransitionZkUtilsExceptionFromCreateStates 
STARTED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransitionZkUtilsExceptionFromCreateStates 
PASSED

kafka.controller.PartitionStateMachineTest > 
testInvalidNewPartitionToNonexistentPartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testInvalidNewPartitionToNonexistentPartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testNewPartitionToOnlinePartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testInvalidOnlinePartitionToNewPartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testInvalidOnlinePartitionToNewPartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testUpdatingOfflinePartitionsCountDuringTopicDeletion STARTED

kafka.controller.PartitionStateMachineTest > 
testUpdatingOfflinePartitionsCountDuringTopicDeletion PASSED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToOnlinePartitionTransitionErrorCodeFromStateLookup STARTED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToOnlinePartitionTransitionErrorCodeFromStateLookup PASSED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOnlineTransitionForControlledShutdown STARTED

kafka.controller.PartitionStateMachineTest > 
testOnlinePartitionToOnlineTransitionForControlledShutdown PASSED

Re: [VOTE] 2.3.0 RC3

[Colin McCabe]

Hi Ismael,

Good catch.  This should be fixed now.

It seems that if the previously staged Sonatype staging repositories (from the 
previous RCs) are not dropped as part of the release process, the new ones 
don't get exposed.  Maybe we should document this somewhere (it was a bit of a 
surprise when I first encountered it.)  Maybe the release.py script should 
mention it.

best,
Colin


On Fri, Jun 21, 2019, at 08:30, Ismael Juma wrote:
> Hi Colin,
> 
> One more thing: the Maven repo was not updated, it seems. Can you fix that?
> 
> Ismael
> 
> On Fri, Jun 21, 2019 at 8:23 AM Ismael Juma  wrote:
> 
> > Hi Colin,
> >
> > Thanks, +1 (binding).
> >
> > I verified the quickstart and signatures for the source and kafka 2.12
> > artifacts with Java 11. One note is that we need to complete the upgrade
> > section for 2.3.0 before we announce the release. No new RC is needed, we
> > just have to have it on the website.
> >
> > Ismael
> >
> > On Wed, Jun 19, 2019 at 3:03 PM Colin McCabe  wrote:
> >
> >> Hi all,
> >>
> >> We discovered some problems with the second release candidate (RC2) of
> >> 2.3.0.  Specifically, KAFKA-8564.  I've created a new RC which includes the
> >> fix for this issue.
> >>
> >> Check out the release notes for the 2.3.0 release here:
> >> https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/RELEASE_NOTES.html
> >>
> >> The vote will go until Saturday, June 22nd, or until we create another RC.
> >>
> >> * Kafka's KEYS file containing PGP keys we use to sign the release can be
> >> found here:
> >> https://kafka.apache.org/KEYS
> >>
> >> * The release artifacts to be voted upon (source and binary) are here:
> >> https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/
> >>
> >> * Maven artifacts to be voted upon:
> >> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >>
> >> * Javadoc:
> >> https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/javadoc/
> >>
> >> * The tag to be voted upon (off the 2.3 branch) is the 2.3.0 tag:
> >> https://github.com/apache/kafka/releases/tag/2.3.0-rc3
> >>
> >> best,
> >> Colin
> >>
> >> C.
> >>
> >
>

[jira] [Created] (KAFKA-8582) Consider adding an ExpiredWindowRecordHandler to Suppress

[John Roesler (JIRA)]

John Roesler created KAFKA-8582:
---

 Summary: Consider adding an ExpiredWindowRecordHandler to Suppress
 Key: KAFKA-8582
 URL: https://issues.apache.org/jira/browse/KAFKA-8582
 Project: Kafka
  Issue Type: Improvement
Reporter: John Roesler


I got some feedback on Suppress:
{quote}Specifying how to handle events outside the grace period does seem like 
a business concern, and simply discarding them thus seems risky (for example 
imagine any situation where money is involved).

This sort of situation is addressed by the late-triggering approach associated 
with watermarks 
(https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102), given 
this I wondered if you were considering adding anything similar?{quote}

It seems like, if a record has arrived past the grace period for its window, 
then the state of the windowed aggregation would already have been lost, so if 
we were to compute an aggregation result, it would be incorrect. Plus, since 
the window is already expired, we can't store the new (incorrect, but more 
importantly expired) aggregation result either, so any subsequent super-late 
records would also face the same blank-slate. I think this would wind up 
looking like this: if you have three timely records for a window, and then 
three more that arrive after the grace period, and you were doing a count 
aggregation, you'd see the counts emitted for the window as [1, 2, 3, 1, 1, 1]. 
I guess we could add a flag to the post-expiration results to indicate that 
they're broken, but this seems like the wrong approach. The post-expiration 
aggregation _results_ are meaningless, but I could see wanting to send the 
past-expiration _input records_ to a dead-letter queue or something instead of 
dropping them.

Along this line of thinking, I wonder if we should add an optional 
past-expiration record handler interface to the suppression operator. Then, you 
could define your own logic, whether it's a dead-letter queue, sending it to 
some alerting pipeline, or even just crashing the application before it can do 
something wrong. This would be a similar pattern to how we allow custom logic 
to handle deserialization errors by supplying a 
org.apache.kafka.streams.errors.DeserializationExceptionHandler.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-8581) Augment ProduceResponse error messaging for specific culprit records

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-8581:


 Summary: Augment ProduceResponse error messaging for specific 
culprit records
 Key: KAFKA-8581
 URL: https://issues.apache.org/jira/browse/KAFKA-8581
 Project: Kafka
  Issue Type: Bug
  Components: clients, core
Reporter: Guozhang Wang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-8106) Reducing the allocation and copying of ByteBuffer when logValidator do validation.

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-8106.
--
   Resolution: Fixed
Fix Version/s: 2.4.0

> Reducing the allocation and copying of ByteBuffer  when logValidator  do 
> validation.
> 
>
> Key: KAFKA-8106
> URL: https://issues.apache.org/jira/browse/KAFKA-8106
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 2.2.0, 2.1.1
> Environment: Server : 
> cpu:2*16 ; 
> MemTotal : 256G;
> Ethernet controller:Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network 
> Connection ; 
> SSD.
>Reporter: Flower.min
>Assignee: Flower.min
>Priority: Major
>  Labels: performance
> Fix For: 2.4.0
>
>
>       We do performance testing about Kafka in specific scenarios as 
> described below .We build a kafka cluster with one broker,and create topics 
> with different number of partitions.Then we start lots of producer processes 
> to send large amounts of messages to one of the topics at one  testing .
> *_Specific Scenario_*
>   
>  *_1.Main config of Kafka_*  
>  # Main config of Kafka  
> server:[num.network.threads=6;num.io.threads=128;queued.max.requests=500|http://num.network.threads%3D6%3Bnum.io.threads%3D128%3Bqueued.max.requests%3D500/]
>  # Number of TopicPartition : 50~2000
>  # Size of Single Message : 1024B
>  
>  *_2.Config of KafkaProducer_* 
> ||compression.type||[linger.ms|http://linger.ms/]||batch.size||buffer.memory||
> |lz4|1000ms~5000ms|16KB/10KB/100KB|128MB|
> *_3.The best result of performance testing_*  
> ||Network inflow rate||CPU Used (%)||Disk write speed||Performance of 
> production||
> |550MB/s~610MB/s|97%～99%|550MB/s~610MB/s       |23,000,000 messages/s|
> *_4.Phenomenon and  my doubt_*
>     _The upper limit of CPU usage has been reached  But  it does not 
> reach the upper limit of the bandwidth of the server  network. *We are 
> doubtful about which  cost too much CPU time and we want to Improve  
> performance and reduces CPU usage of Kafka server.*_
> _*5.Analysis*_
>         We analysis the JFIR of Kafka server when doing performance testing 
> .After we checked and completed the performance test again, we located the 
> code "*ByteBuffer recordBuffer = 
> ByteBuffer.allocate(sizeOfBodyInBytes);*(*Class:DefaultRecord,Function:readFrom()*)”
>  which consumed CPU resources and caused a lot of GC .Our modified code 
> reduces the allocation and copying of ByteBuffer, so the test performance is 
> greatly improved, and the CPU's stable usage is *below 60%*. The following is 
> a comparison of different code test performance under the same conditions.
> *Result of performance testing*
> *Main config of Kafka: Single 
> Message:1024B;TopicPartitions:200;linger.ms:1000ms.*
> | Single Message : 1024B,|Network inflow rate|CPU(%)|Messages/s|
> |Source code|600M/s|97%|25,000,000|
> |Modified code|1GB/s|<60%|41,660,000|
> **1.Before modified code(Source code) GC:**
> ![](https://i.loli.net/2019/05/07/5cd16df163ad3.png)
> **2.After modified code(remove allocation of ByteBuffer) GC:**
> ![](https://i.loli.net/2019/05/07/5cd16dae1dbc2.png)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk11 #649

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-8519 Add trogdor action to slow down a network (#6912)

--
[...truncated 2.52 MB...]

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testLeaderPerformAssignment1[1] STARTED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testLeaderPerformAssignment1[1] PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testLeaderPerformAssignment2[1] STARTED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testLeaderPerformAssignment2[1] PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testJoinLeaderCannotAssign[1] STARTED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testJoinLeaderCannotAssign[1] PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testRejoinGroup[1] STARTED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testRejoinGroup[1] PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testNormalJoinGroupLeader[1] STARTED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testNormalJoinGroupLeader[1] PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorIncrementalTest > 
testMetadataWithExistingAssignmentButOlderProtocolSelection[0] STARTED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorIncrementalTest > 
testMetadataWithExistingAssignmentButOlderProtocolSelection[0] PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorIncrementalTest > 
testMetadataWithExistingAssignment[0] STARTED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorIncrementalTest > 
testMetadataWithExistingAssignment[0] PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorIncrementalTest > 
testTaskAssignmentWhenWorkerBounces[0] STARTED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorIncrementalTest > 
testTaskAssignmentWhenWorkerBounces[0] PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorIncrementalTest > 
testTaskAssignmentWhenWorkerJoins[0] STARTED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorIncrementalTest > 
testTaskAssignmentWhenWorkerJoins[0] PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorIncrementalTest > 
testTaskAssignmentWhenWorkerLeavesPermanently[0] STARTED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorIncrementalTest > 
testTaskAssignmentWhenWorkerLeavesPermanently[0] PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorIncrementalTest > 
testMetadata[0] STARTED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorIncrementalTest > 
testMetadata[0] PASSED

org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignorTest 
> testLostAssignmentHandlingWhenWorkerLeavesPermanently STARTED

org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignorTest 
> testLostAssignmentHandlingWhenWorkerLeavesPermanently PASSED

org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignorTest 
> testTaskAssignmentWhenLeaderBounces STARTED

org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignorTest 
> testTaskAssignmentWhenLeaderBounces PASSED

org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignorTest 
> testTaskAssignmentWhenWorkerBounces STARTED

org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignorTest 
> testTaskAssignmentWhenWorkerBounces PASSED

org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignorTest 
> testLostAssignmentHandlingWhenWorkerBouncesBackButFinallyLeaves STARTED

org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignorTest 
> testLostAssignmentHandlingWhenWorkerBouncesBackButFinallyLeaves PASSED

org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignorTest 
> testLostAssignmentHandlingWhenWorkerBounces STARTED

org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignorTest 
> testLostAssignmentHandlingWhenWorkerBounces PASSED

org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignorTest 
> testTaskAssignmentWhenWorkerJoins STARTED

org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignorTest 
> testTaskAssignmentWhenWorkerJoins PASSED

org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignorTest 
> testLostAssignmentHandlingWithMoreThanOneCandidates STARTED

org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignorTest 
> testLostAssignmentHandlingWithMoreThanOneCandidates PASSED

org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignorTest 
> testAssignTasksWhenBalanced STARTED

org.apache.kafka.connect.runtime.distributed.IncrementalCooperativeAssignorTest 
> 

Re: [DISCUSS] KIP-470: TopologyTestDriver test input and output usability improvements

[Matthias J. Sax]

Thanks for the KIP. The idea to add InputTopic and OutputTopic
abstractions is really neat!


Couple of minor comment:

(1) It's a little confusing that you list all method (existing, proposed
to deprecate, and new one) of `TopologyTestDriver` in the KIP. Maybe
only list the ones you propose to deprecate and the new ones you want to
add?

(Or mark all existing methods clearly -- atm, I need to got back to the
code to read the KIP and to extract what changes are proposed).


(2) `TopologyTestDriver#createInputTopic`: might it be worth to add
overload to initialize the timetamp and auto-advance feature directly?
Otherwise, uses always need to call `configureTiming` as an extra call?


(3) `TestInputTopic#configureTiming()`: maybe rename to
`reconfigureTiming()` ?


(4) Should we switch from `long` for timestamps to `Instant` and
`Duration` ?


(5) Why do we have redundant getters? Or set with `getX()` and one set
without `get`-prefix?



-Matthias




On 6/21/19 10:57 AM, Bill Bejeck wrote:
> Jukka,
> 
> Thanks for the KIP. I like the changes overall.
> One thing I wanted to confirm, and this may be me being paranoid, but will
> the changes for input/output topic affect how the TopologyTestDriver works
> with internal topics when there are sub-topologies created?
> 
> On Fri, Jun 21, 2019 at 12:05 PM Guozhang Wang  wrote:
> 
>> 1) Got it, could you list this class along with all its functions in the
>> proposed public APIs as well?
>>
>> 2) Ack, thanks!
>>
>> On Thu, Jun 20, 2019 at 11:27 PM Jukka Karvanen <
>> jukka.karva...@jukinimi.com>
>> wrote:
>>
>>> Hi  Guozhang,
>>>
>>> 1) This TestRecord is new class in my proposal. So it is a simplified
>>> version of ProducerRecord and ConsumerRecord containing only the fields
>>> needed to test record content.
>>>
>>> 2)
>>> public final  TestInputTopic createInputTopic(final String
>>> topicName, final Serde keySerde, final Serde valueSerde);
>>> public final  TestOutputTopic createOutputTopic(final String
>>> topicName, final Serde keySerde, final Serde valueSerde);
>>> The purpose is to create separate object for each input and output topic
>>> you are using. The topic name is given to createInput/OutputTopic when
>>> initialize topic object.
>>>
>>> For example:
>>>
>>> final TestInputTopic inputTopic1 =
>>> testDriver.createInputTopic( INPUT_TOPIC, longSerde, stringSerde);
>>> final TestInputTopic inputTopic2 =
>>> testDriver.createInputTopic( INPUT_TOPIC_MAP, longSerde, stringSerde);
>>> final TestOutputTopic outputTopic1 =
>>> testDriver.createOutputTopic(OUTPUT_TOPIC, longSerde, stringSerde);
>>> final TestOutputTopic outputTopic2 =
>>> testDriver.createOutputTopic(OUTPUT_TOPIC_MAP, stringSerde,
>>> longSerde);
>>> inputTopic1.pipeInput(1L, "Hello");
>>> assertThat(outputTopic1.readKeyValue(), equalTo(new KeyValue<>(1L,
>>> "Hello")));
>>> assertThat(outputTopic2.readKeyValue(), equalTo(new KeyValue<>("Hello",
>>> 1L)));
>>> inputTopic2.pipeInput(1L, "Hello");
>>>
>>>
>>> Jukka
>>>
>>> to 20. kesäk. 2019 klo 23.52 Guozhang Wang (wangg...@gmail.com)
>> kirjoitti:
>>>
 Hello Jukka,

 Thanks for writing the KIP, I have a couple of quick questions:

 1) Is "TestRecord" an existing class that you propose to piggy-back on?
 Right now we have a scala TestRecord case class but I doubt that was
>> your
 proposal, or are you proposing to add a new Java class?

 2) Would the new API only allow a single input / output topic with
 `createInput/OutputTopic`? If not, when we call pipeInput how to
>>> determine
 which topic this record should be pipe to?


 Guozhang

 On Mon, Jun 17, 2019 at 1:34 PM John Roesler 
>> wrote:

> Woah, I wasn't aware of that Hamcrest test style. Awesome!
>
> Thanks for the updates. I look forward to hearing what others think.
>
> -John
>
> On Mon, Jun 17, 2019 at 4:12 AM Jukka Karvanen
>  wrote:
>>
>> Wiki page updated:
>>
>

>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-470%3A+TopologyTestDriver+test+input+and+output+usability+improvements
>>
>>
>> ClientRecord removed and replaced with TestRecord in method calls.
>> TestRecordFactory removed (time tracking functionality to be
>> included
 to
>> TestInputTopic)
>> OutputVerifier deprecated
>> TestRecord topic removed and getters added
>>
>> Getters in TestRecord enable writing test ignoring selected fields
>>> with
>> hamcrest like this:
>>
>> assertThat(outputTopic.readRecord(), allOf(
>> hasProperty("key", equalTo(1L)),
>> hasProperty("value", equalTo("Hello")),
>> hasProperty("headers", equalTo(headers;
>>
>>
>> Jukka
>>
>> la 15. kesäk. 2019 klo 1.10 John Roesler (j...@confluent.io)
 kirjoitti:
>>
>>> Sounds good. Thanks as always for considering my feedback!
>>> -John
>>>
>>> On Fri, Jun 14, 2019 at 12:12 PM Jukka 

Build failed in Jenkins: kafka-trunk-jdk8 #3741

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-8519 Add trogdor action to slow down a network (#6912)

--
[...truncated 2.51 MB...]
org.apache.kafka.streams.scala.WordCountTest > testShouldCountWordsMaterialized 
STARTED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWordsMaterialized 
PASSED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWordsJava STARTED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWordsJava PASSED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWords STARTED

org.apache.kafka.streams.scala.WordCountTest > testShouldCountWords PASSED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialized 
should create a Materialized with Serdes STARTED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialized 
should create a Materialized with Serdes PASSED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a store name should create a Materialized with Serdes and a store name 
STARTED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a store name should create a Materialized with Serdes and a store name 
PASSED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a window store supplier should create a Materialized with Serdes and a 
store supplier STARTED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a window store supplier should create a Materialized with Serdes and a 
store supplier PASSED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a key value store supplier should create a Materialized with Serdes and a 
store supplier STARTED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a key value store supplier should create a Materialized with Serdes and a 
store supplier PASSED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a session store supplier should create a Materialized with Serdes and a 
store supplier STARTED

org.apache.kafka.streams.scala.kstream.MaterializedTest > Create a Materialize 
with a session store supplier should create a Materialized with Serdes and a 
store supplier PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > filter a KTable should 
filter records satisfying the predicate STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > filter a KTable should 
filter records satisfying the predicate PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > filterNot a KTable should 
filter records not satisfying the predicate STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > filterNot a KTable should 
filter records not satisfying the predicate PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables should join 
correctly records STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables should join 
correctly records PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables with a 
Materialized should join correctly records and state store STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > join 2 KTables with a 
Materialized should join correctly records and state store PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilTimeLimit STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilTimeLimit PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilWindowCloses STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > windowed KTable#suppress 
should correctly suppress results using Suppressed.untilWindowCloses PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > session windowed 
KTable#suppress should correctly suppress results using 
Suppressed.untilWindowCloses STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > session windowed 
KTable#suppress should correctly suppress results using 
Suppressed.untilWindowCloses PASSED

org.apache.kafka.streams.scala.kstream.KTableTest > non-windowed 
KTable#suppress should correctly suppress results using 
Suppressed.untilTimeLimit STARTED

org.apache.kafka.streams.scala.kstream.KTableTest > non-windowed 
KTable#suppress should correctly suppress results using 
Suppressed.untilTimeLimit PASSED

org.apache.kafka.streams.scala.kstream.SuppressedTest > 
Suppressed.untilWindowCloses should produce the correct suppression STARTED

org.apache.kafka.streams.scala.kstream.SuppressedTest > 
Suppressed.untilWindowCloses should produce the correct suppression PASSED

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

[Matthias J. Sax]

Thanks for the update!

I think that `ListDeserializer`, `ListSerializer`, and `ListSerde`
should have an default constructor and it should be possible to pass in
the `Class listClass` information via a configuration. Otherwise,
KafkaStreams cannot use it as default serde.


For the primitive serializers: `BytesSerializer` is not primitive IMHO,
as is it for `byte[]` with variable length -- it's for arrays, not for
single `byte` (note, that `Bytes` is a Kafka class wrapping `byte[]`).


For tests, we can comment on the PR. No need to do this in the KIP
discussion.


Can you also update the KIP?



-Matthias





On 6/21/19 11:29 AM, Development wrote:
> I made and pushed necessary commits, so we could review the final version 
> under PR https://github.com/apache/kafka/pull/6592
> 
> I also need some advice on writing tests for this new serde. So far I only 
> have two test cases (roundtrip and empty payload), I’m not sure if it is 
> enough.
> 
> Thank y’all for your help in this KIP :)
> 
> Best,
> Daniyar Yeralin
> 
> 
>> On Jun 21, 2019, at 1:44 PM, John Roesler  wrote:
>>
>> Hey Daniyar,
>>
>> Looks good to me! Thanks for considering it.
>>
>> Thanks,
>> -John
>>
>> On Fri, Jun 21, 2019 at 9:04 AM Development > > wrote:
>> Hey John and Matthias,
>>
>> Yes, now I see it all. I’m storing lots of redundant information.
>> Here is my final idea. Yes, now a user should pass a list type. I realized 
>> that’s the type is not really needed in ListSerializer, but only in 
>> ListDeserializer:
>>
>>
>> In ListSerializer we will start storing sizes only if serializer is not a 
>> primitive serializer:
>>
>>
>> Then, in deserializer, we persist passed list type, so that during 
>> deserialization we could create an instance of it with predefined listSize 
>> for better performance.
>> We also try to locate a primitiveSize based on passed deserializer. If it is 
>> not there, then primitiveSize will be null. Which means that each entry’s 
>> size was encoded individually.
>>
>>
>> This looks much cleaner and more concise.
>>
>> What do you think?
>>
>> Best,
>> Daniyar Yeralin 
>>
>>> On Jun 20, 2019, at 5:45 PM, Matthias J. Sax >> > wrote:
>>>
>>> For encoding the list-type: I see John's point about re-encoding the
>>> list-type redundantly. However, I also don't like the idea that the
>>> Deserializer returns a fixed type...
>>>
>>> Maybe it's best allow users to specify the target list type on
>>> deserialization via config?
>>>
>>> Similar for the primitive types: I don't think we need to encode the
>>> type size, but users could specify the type on the deserializer (via a
>>> config again)?
>>>
>>>
>>> About generics: nesting could be arbitrarily deep. Hence, I doubt we can
>>> support this and a cast will be necessary at some point in the user code.
>>>
>>>
>>>
>>> -Matthias
>>>
>>>
>>>
>>> On 6/20/19 1:21 PM, John Roesler wrote:
 Hey Daniyar,

 Thanks for looking at it!

 Something like your screenshot is more along the lines of what I was
 thinking. Sorry, but I didn't follow what you mean, how would that not
 be "vanilla java"?

 Unfortunately the deserializer needs more information, though. For
 example, what if the inner type is a Map? The serde could
 only be used to produce a LinkedList, thus, we'd still need an
 inner serde, like you have in the KIP (Serde innerSerde).

 Something more like Serde> = Serdes.listSerde(
   /**list type**/ LinkedList.class,
   /**inner serde**/ new MyRecordSerde()
 )

 And in configuration, it's something like:
 default.key.serde: org...ListSerde
 default.key.list.serde.type: java.util.LinkedList
 default.key.list.serde.inner: com.mycompany.MyRecordSerde


 What do you think?
 Thanks,
 -John

 On Thu, Jun 20, 2019 at 2:46 PM Development >>> 
 >> wrote:

Hey John,

I gave read about TypeReference. It could work for the list serde.
However, it is not directly
supported: https://github.com/FasterXML/jackson-databind/issues/1490 
 
The only way is to pass an actual class object into the constructor,
something like:

It could be an option, but not a pretty one. What do you think of my
approach to use vanilla java and canonical class name? (As described
previously)

Best,
Daniyar Yeralin

>On Jun 20, 2019, at 2:45 PM, Development  
>>> wrote:
>
>Hi John,
>
>Thank you for your input! Yes, my idea looks a little bit over
>engineered :)
>
>I also wanted to see a feedback from Mathias as well since he gave
>me an idea 

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

[Development]

I made and pushed necessary commits, so we could review the final version under 
PR https://github.com/apache/kafka/pull/6592

I also need some advice on writing tests for this new serde. So far I only have 
two test cases (roundtrip and empty payload), I’m not sure if it is enough.

Thank y’all for your help in this KIP :)

Best,
Daniyar Yeralin


> On Jun 21, 2019, at 1:44 PM, John Roesler  wrote:
> 
> Hey Daniyar,
> 
> Looks good to me! Thanks for considering it.
> 
> Thanks,
> -John
> 
> On Fri, Jun 21, 2019 at 9:04 AM Development  > wrote:
> Hey John and Matthias,
> 
> Yes, now I see it all. I’m storing lots of redundant information.
> Here is my final idea. Yes, now a user should pass a list type. I realized 
> that’s the type is not really needed in ListSerializer, but only in 
> ListDeserializer:
> 
> 
> In ListSerializer we will start storing sizes only if serializer is not a 
> primitive serializer:
> 
> 
> Then, in deserializer, we persist passed list type, so that during 
> deserialization we could create an instance of it with predefined listSize 
> for better performance.
> We also try to locate a primitiveSize based on passed deserializer. If it is 
> not there, then primitiveSize will be null. Which means that each entry’s 
> size was encoded individually.
> 
> 
> This looks much cleaner and more concise.
> 
> What do you think?
> 
> Best,
> Daniyar Yeralin 
> 
>> On Jun 20, 2019, at 5:45 PM, Matthias J. Sax > > wrote:
>> 
>> For encoding the list-type: I see John's point about re-encoding the
>> list-type redundantly. However, I also don't like the idea that the
>> Deserializer returns a fixed type...
>> 
>> Maybe it's best allow users to specify the target list type on
>> deserialization via config?
>> 
>> Similar for the primitive types: I don't think we need to encode the
>> type size, but users could specify the type on the deserializer (via a
>> config again)?
>> 
>> 
>> About generics: nesting could be arbitrarily deep. Hence, I doubt we can
>> support this and a cast will be necessary at some point in the user code.
>> 
>> 
>> 
>> -Matthias
>> 
>> 
>> 
>> On 6/20/19 1:21 PM, John Roesler wrote:
>>> Hey Daniyar,
>>> 
>>> Thanks for looking at it!
>>> 
>>> Something like your screenshot is more along the lines of what I was
>>> thinking. Sorry, but I didn't follow what you mean, how would that not
>>> be "vanilla java"?
>>> 
>>> Unfortunately the deserializer needs more information, though. For
>>> example, what if the inner type is a Map? The serde could
>>> only be used to produce a LinkedList, thus, we'd still need an
>>> inner serde, like you have in the KIP (Serde innerSerde).
>>> 
>>> Something more like Serde> = Serdes.listSerde(
>>>   /**list type**/ LinkedList.class,
>>>   /**inner serde**/ new MyRecordSerde()
>>> )
>>> 
>>> And in configuration, it's something like:
>>> default.key.serde: org...ListSerde
>>> default.key.list.serde.type: java.util.LinkedList
>>> default.key.list.serde.inner: com.mycompany.MyRecordSerde
>>> 
>>> 
>>> What do you think?
>>> Thanks,
>>> -John
>>> 
>>> On Thu, Jun 20, 2019 at 2:46 PM Development >> 
>>> >> wrote:
>>> 
>>>Hey John,
>>> 
>>>I gave read about TypeReference. It could work for the list serde.
>>>However, it is not directly
>>>supported: https://github.com/FasterXML/jackson-databind/issues/1490 
>>> 
>>>The only way is to pass an actual class object into the constructor,
>>>something like:
>>> 
>>>It could be an option, but not a pretty one. What do you think of my
>>>approach to use vanilla java and canonical class name? (As described
>>>previously)
>>> 
>>>Best,
>>>Daniyar Yeralin
>>> 
On Jun 20, 2019, at 2:45 PM, Development >>> 
>> wrote:
 
Hi John,
 
Thank you for your input! Yes, my idea looks a little bit over
engineered :)
 
I also wanted to see a feedback from Mathias as well since he gave
me an idea about storing fixed/variable size entries.
 
Best,
Daniyar Yeralin
 
>On Jun 18, 2019, at 6:06 PM, John Roesler  
>>> wrote:
> 
>Hi Daniyar,
> 
>That's a very clever solution!
> 
>One observation is that, now, this is what we might call a
>polymorphic
>serde. That is, you're detecting the actual concrete type and then
>promising to produce the exact same concrete type on read. There are
>some inherent problems with this approach, which in general require
>some kind of  schema registry (not necessarily Schema Registry, just
>any registry for schemas) to solve.

Re: [DISCUSS] KIP-470: TopologyTestDriver test input and output usability improvements

[Bill Bejeck]

Jukka,

Thanks for the KIP. I like the changes overall.
One thing I wanted to confirm, and this may be me being paranoid, but will
the changes for input/output topic affect how the TopologyTestDriver works
with internal topics when there are sub-topologies created?

On Fri, Jun 21, 2019 at 12:05 PM Guozhang Wang  wrote:

> 1) Got it, could you list this class along with all its functions in the
> proposed public APIs as well?
>
> 2) Ack, thanks!
>
> On Thu, Jun 20, 2019 at 11:27 PM Jukka Karvanen <
> jukka.karva...@jukinimi.com>
> wrote:
>
> > Hi  Guozhang,
> >
> > 1) This TestRecord is new class in my proposal. So it is a simplified
> > version of ProducerRecord and ConsumerRecord containing only the fields
> > needed to test record content.
> >
> > 2)
> > public final  TestInputTopic createInputTopic(final String
> > topicName, final Serde keySerde, final Serde valueSerde);
> > public final  TestOutputTopic createOutputTopic(final String
> > topicName, final Serde keySerde, final Serde valueSerde);
> > The purpose is to create separate object for each input and output topic
> > you are using. The topic name is given to createInput/OutputTopic when
> > initialize topic object.
> >
> > For example:
> >
> > final TestInputTopic inputTopic1 =
> > testDriver.createInputTopic( INPUT_TOPIC, longSerde, stringSerde);
> > final TestInputTopic inputTopic2 =
> > testDriver.createInputTopic( INPUT_TOPIC_MAP, longSerde, stringSerde);
> > final TestOutputTopic outputTopic1 =
> > testDriver.createOutputTopic(OUTPUT_TOPIC, longSerde, stringSerde);
> > final TestOutputTopic outputTopic2 =
> > testDriver.createOutputTopic(OUTPUT_TOPIC_MAP, stringSerde,
> > longSerde);
> > inputTopic1.pipeInput(1L, "Hello");
> > assertThat(outputTopic1.readKeyValue(), equalTo(new KeyValue<>(1L,
> > "Hello")));
> > assertThat(outputTopic2.readKeyValue(), equalTo(new KeyValue<>("Hello",
> > 1L)));
> > inputTopic2.pipeInput(1L, "Hello");
> >
> >
> > Jukka
> >
> > to 20. kesäk. 2019 klo 23.52 Guozhang Wang (wangg...@gmail.com)
> kirjoitti:
> >
> > > Hello Jukka,
> > >
> > > Thanks for writing the KIP, I have a couple of quick questions:
> > >
> > > 1) Is "TestRecord" an existing class that you propose to piggy-back on?
> > > Right now we have a scala TestRecord case class but I doubt that was
> your
> > > proposal, or are you proposing to add a new Java class?
> > >
> > > 2) Would the new API only allow a single input / output topic with
> > > `createInput/OutputTopic`? If not, when we call pipeInput how to
> > determine
> > > which topic this record should be pipe to?
> > >
> > >
> > > Guozhang
> > >
> > > On Mon, Jun 17, 2019 at 1:34 PM John Roesler 
> wrote:
> > >
> > > > Woah, I wasn't aware of that Hamcrest test style. Awesome!
> > > >
> > > > Thanks for the updates. I look forward to hearing what others think.
> > > >
> > > > -John
> > > >
> > > > On Mon, Jun 17, 2019 at 4:12 AM Jukka Karvanen
> > > >  wrote:
> > > > >
> > > > > Wiki page updated:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-470%3A+TopologyTestDriver+test+input+and+output+usability+improvements
> > > > >
> > > > >
> > > > > ClientRecord removed and replaced with TestRecord in method calls.
> > > > > TestRecordFactory removed (time tracking functionality to be
> included
> > > to
> > > > > TestInputTopic)
> > > > > OutputVerifier deprecated
> > > > > TestRecord topic removed and getters added
> > > > >
> > > > > Getters in TestRecord enable writing test ignoring selected fields
> > with
> > > > > hamcrest like this:
> > > > >
> > > > > assertThat(outputTopic.readRecord(), allOf(
> > > > > hasProperty("key", equalTo(1L)),
> > > > > hasProperty("value", equalTo("Hello")),
> > > > > hasProperty("headers", equalTo(headers;
> > > > >
> > > > >
> > > > > Jukka
> > > > >
> > > > > la 15. kesäk. 2019 klo 1.10 John Roesler (j...@confluent.io)
> > > kirjoitti:
> > > > >
> > > > > > Sounds good. Thanks as always for considering my feedback!
> > > > > > -John
> > > > > >
> > > > > > On Fri, Jun 14, 2019 at 12:12 PM Jukka Karvanen
> > > > > >  wrote:
> > > > > > >
> > > > > > > Ok, I will modify KIP Public Interface in a wiki based on the
> > > > feedback.
> > > > > > >
> > > > > > > TestRecordFactory / ConsumerRecordFactory was used by
> > > TestInputTopic
> > > > with
> > > > > > > the version I had with KIP456, but maybe I can merge That
> > > > functionality
> > > > > > to
> > > > > > > InputTopic or  TestRecordFactory   can kept non public maybe
> > moving
> > > > it to
> > > > > > > internals package.
> > > > > > >
> > > > > > > I will make the proposal with a slim down interface.
> > > > > > > I don't want to go to so slim as you proposed with only
> > TestRecord
> > > or
> > > > > > > List, because you then still end up doing helper
> > > methods
> > > > to
> > > > > > > construct List of TestRecord.
> > > > > > > The list of values is easier to write and clearer to read than
> if
> > > you
> > > > > > need
> > 

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

[John Roesler]

Hey Daniyar,

Looks good to me! Thanks for considering it.

Thanks,
-John

On Fri, Jun 21, 2019 at 9:04 AM Development  wrote:

> Hey John and Matthias,
>
> Yes, now I see it all. I’m storing lots of redundant information.
> Here is my final idea. Yes, now a user should pass a list type. I realized
> that’s the type is not really needed in ListSerializer, but only in
> ListDeserializer:
>
> In ListSerializer we will start storing sizes only if serializer is not a
> primitive serializer:
>
> Then, in deserializer, we persist passed list type, so that during
> deserialization we could create an instance of it with predefined listSize
> for better performance.
> We also try to locate a primitiveSize based on passed deserializer. If it
> is not there, then primitiveSize will be null. Which means that each
> entry’s size was encoded individually.
>
> This looks much cleaner and more concise.
>
> What do you think?
>
> Best,
> Daniyar Yeralin
>
> On Jun 20, 2019, at 5:45 PM, Matthias J. Sax 
> wrote:
>
> For encoding the list-type: I see John's point about re-encoding the
> list-type redundantly. However, I also don't like the idea that the
> Deserializer returns a fixed type...
>
> Maybe it's best allow users to specify the target list type on
> deserialization via config?
>
> Similar for the primitive types: I don't think we need to encode the
> type size, but users could specify the type on the deserializer (via a
> config again)?
>
>
> About generics: nesting could be arbitrarily deep. Hence, I doubt we can
> support this and a cast will be necessary at some point in the user code.
>
>
>
> -Matthias
>
>
>
> On 6/20/19 1:21 PM, John Roesler wrote:
>
> Hey Daniyar,
>
> Thanks for looking at it!
>
> Something like your screenshot is more along the lines of what I was
> thinking. Sorry, but I didn't follow what you mean, how would that not
> be "vanilla java"?
>
> Unfortunately the deserializer needs more information, though. For
> example, what if the inner type is a Map? The serde could
> only be used to produce a LinkedList, thus, we'd still need an
> inner serde, like you have in the KIP (Serde innerSerde).
>
> Something more like Serde> = Serdes.listSerde(
>   /**list type**/ LinkedList.class,
>   /**inner serde**/ new MyRecordSerde()
> )
>
> And in configuration, it's something like:
> default.key.serde: org...ListSerde
> default.key.list.serde.type: java.util.LinkedList
> default.key.list.serde.inner: com.mycompany.MyRecordSerde
>
>
> What do you think?
> Thanks,
> -John
>
> On Thu, Jun 20, 2019 at 2:46 PM Development  > wrote:
>
>Hey John,
>
>I gave read about TypeReference. It could work for the list serde.
>However, it is not directly
>supported: https://github.com/FasterXML/jackson-databind/issues/1490
>The only way is to pass an actual class object into the constructor,
>something like:
>
>It could be an option, but not a pretty one. What do you think of my
>approach to use vanilla java and canonical class name? (As described
>previously)
>
>Best,
>Daniyar Yeralin
>
>On Jun 20, 2019, at 2:45 PM, Development > wrote:
>
>Hi John,
>
>Thank you for your input! Yes, my idea looks a little bit over
>engineered :)
>
>I also wanted to see a feedback from Mathias as well since he gave
>me an idea about storing fixed/variable size entries.
>
>Best,
>Daniyar Yeralin
>
>On Jun 18, 2019, at 6:06 PM, John Roesler > wrote:
>
>Hi Daniyar,
>
>That's a very clever solution!
>
>One observation is that, now, this is what we might call a
>polymorphic
>serde. That is, you're detecting the actual concrete type and then
>promising to produce the exact same concrete type on read. There are
>some inherent problems with this approach, which in general require
>some kind of  schema registry (not necessarily Schema Registry, just
>any registry for schemas) to solve.
>
>Notice that every serialized record has quite a bit of duplicated
>information: the concrete type as well as a byte to indicate whether
>the value type is a fixed size, and, if so, an integer to
>indicate the
>actual size. These constitute a schema, of sorts, because they
>tell us
>later how exactly to deserialize the data. Unfortunately, this
>information is completely redundant. In all likelihood, the
>information will be exactly the same for every record in the topic.
>This problem is essentially the core motivation for serializations
>like Avro: to move the schema outside of the serialization itself, so
>that the records won't contain so much redundant information.
>
>In this light, I'm wondering if it makes sense to go back to
>something
>like what you had earlier in which you don't support perfectly
>preserving the concrete type for _this_ serde, but instead just
>support deserializing to 

[jira] [Resolved] (KAFKA-8563) Minor: Remove method call in networkSend. Rely on java's vargs boxing/autoboxing

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-8563.

   Resolution: Fixed
Fix Version/s: 2.4.0

> Minor: Remove method call in networkSend. Rely on java's vargs 
> boxing/autoboxing
> 
>
> Key: KAFKA-8563
> URL: https://issues.apache.org/jira/browse/KAFKA-8563
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 2.4.0
> Environment: Darwin WM-CXX 18.2.0 Darwin Kernel Version 18.2.0: 
> Thu Dec 20 20:46:53 PST 2018; root:xnu-4903.241.1~1/RELEASE_X86_64 x86_64
> ProductName:  Mac OS X
> ProductVersion:   10.14.3
> java version "1.8.0_201"
> Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
> Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)
>Reporter: karan kumar
>Priority: Minor
> Fix For: 2.4.0
>
>
> There was a  
> [https://github.com/apache/kafka/blob/93bf96589471acadfb90e57ebfecbd91f679f77b/clients/src/main/java/org/apache/kafka/common/network/NetworkSend.java#L30]
>  which can be removed from the network send class. 
>  
> Initial JMH benchmarks suggest no performance penalty.
>  
> Present network send JMH report:
>  
> {code:java}
> jmh-benchmarks git:(trunk) ✗ ./jmh.sh -f 2 ByteBufferSendBenchmark
> running gradlew :jmh-benchmarks:clean :jmh-benchmarks:shadowJar in quiet mode
> ./jmh.sh: line 34: ../gradlew: No such file or directory
> gradle build done
> running JMH with args [-f 2 ByteBufferSendBenchmark]
> # JMH version: 1.21
> # VM version: JDK 1.8.0_201, Java HotSpot(TM) 64-Bit Server VM, 25.201-b09
> # VM invoker: 
> /Library/Java/JavaVirtualMachines/jdk1.8.0_201.jdk/Contents/Home/jre/bin/java
> # VM options: 
> # Warmup: 5 iterations, 2000 ms each
> # Measurement: 5 iterations, 5000 ms each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: 
> org.apache.kafka.jmh.common.ByteBufferSendBenchmark.benchmarkMethod
> # Run progress: 0.00% complete, ETA 00:01:10
> # Fork: 1 of 2
> # Warmup Iteration 1: 35.049 ops/us
> # Warmup Iteration 2: 60.877 ops/us
> # Warmup Iteration 3: 59.207 ops/us
> # Warmup Iteration 4: 59.077 ops/us
> # Warmup Iteration 5: 59.327 ops/us
> Iteration 1: 58.516 ops/us
> Iteration 2: 58.952 ops/us
> Iteration 3: 58.596 ops/us
> Iteration 4: 59.126 ops/us
> Iteration 5: 58.557 ops/us
> # Run progress: 50.00% complete, ETA 00:00:35
> # Fork: 2 of 2
> # Warmup Iteration 1: 36.377 ops/us
> # Warmup Iteration 2: 61.741 ops/us
> # Warmup Iteration 3: 59.683 ops/us
> # Warmup Iteration 4: 59.571 ops/us
> # Warmup Iteration 5: 59.351 ops/us
> Iteration 1: 59.044 ops/us
> Iteration 2: 59.107 ops/us
> Iteration 3: 57.771 ops/us
> Iteration 4: 59.648 ops/us
> Iteration 5: 59.408 ops/us
> Result "org.apache.kafka.jmh.common.ByteBufferSendBenchmark.benchmarkMethod":
> 58.872 ±(99.9%) 0.806 ops/us [Average]
> (min, avg, max) = (57.771, 58.872, 59.648), stdev = 0.533
> CI (99.9%): [58.066, 59.679] (assumes normal distribution)
> # Run complete. Total time: 00:01:11
> REMEMBER: The numbers below are just data. To gain reusable insights, you 
> need to follow up on
> why the numbers are the way they are. Use profilers (see -prof, -lprof), 
> design factorial
> experiments, perform baseline and negative tests that provide experimental 
> control, make sure
> the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from 
> the domain experts.
> Do not assume the numbers tell you what you want them to tell.
> Benchmark Mode Cnt Score Error Units
> *ByteBufferSendBenchmark.benchmarkMethod thrpt 10 58.872 ± 0.806 ops/us*
> JMH benchmarks done
> {code}
> and after removing the method call
>  
> {code:java}
> // code placeholder
> ./jmh.sh: line 34: ../gradlew: No such file or directory
> gradle build done
> running JMH with args [-f 2 ByteBufferSendBenchmark]
> # JMH version: 1.21
> # VM version: JDK 1.8.0_201, Java HotSpot(TM) 64-Bit Server VM, 25.201-b09
> # VM invoker: 
> /Library/Java/JavaVirtualMachines/jdk1.8.0_201.jdk/Contents/Home/jre/bin/java
> # VM options: 
> # Warmup: 5 iterations, 2000 ms each
> # Measurement: 5 iterations, 5000 ms each
> # Timeout: 10 min per iteration
> # Threads: 1 thread, will synchronize iterations
> # Benchmark mode: Throughput, ops/time
> # Benchmark: 
> org.apache.kafka.jmh.common.ByteBufferSendBenchmark.benchmarkMethod
> # Run progress: 0.00% complete, ETA 00:01:10
> # Fork: 1 of 2
> # Warmup Iteration 1: 34.273 ops/us
> # Warmup Iteration 2: 61.565 ops/us
> # Warmup Iteration 3: 59.307 ops/us
> # Warmup Iteration 4: 57.081 ops/us
> # Warmup Iteration 5: 59.970 ops/us
> Iteration 1: 59.657 ops/us
> Iteration 2: 59.607 ops/us
> Iteration 3: 59.931 ops/us
> 

[jira] [Resolved] (KAFKA-8570) Downconversion could fail when log contains out of order message formats

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-8570.

   Resolution: Fixed
Fix Version/s: 2.2.2
   2.1.2
   2.0.2
   2.3.1

> Downconversion could fail when log contains out of order message formats
> 
>
> Key: KAFKA-8570
> URL: https://issues.apache.org/jira/browse/KAFKA-8570
> Project: Kafka
>  Issue Type: Bug
>Reporter: Dhruvil Shah
>Assignee: Dhruvil Shah
>Priority: Major
> Fix For: 2.3.1, 2.0.2, 2.1.2, 2.2.2
>
>
> When the log contains out of order message formats (for example a v2 message 
> followed by a v1 message), it is possible for down-conversion to fail in 
> certain scenarios where batches compressed and greater than 1kB in size. 
> Down-conversion fails with a stack like the following:
> java.lang.IllegalArgumentException
> at java.nio.Buffer.limit(Buffer.java:275)
> at 
> org.apache.kafka.common.record.FileLogInputStream$FileChannelRecordBatch.writeTo(FileLogInputStream.java:176)
> at 
> org.apache.kafka.common.record.AbstractRecords.downConvert(AbstractRecords.java:107)
> at 
> org.apache.kafka.common.record.FileRecords.downConvert(FileRecords.java:242)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-470: TopologyTestDriver test input and output usability improvements

[Guozhang Wang]

1) Got it, could you list this class along with all its functions in the
proposed public APIs as well?

2) Ack, thanks!

On Thu, Jun 20, 2019 at 11:27 PM Jukka Karvanen 
wrote:

> Hi  Guozhang,
>
> 1) This TestRecord is new class in my proposal. So it is a simplified
> version of ProducerRecord and ConsumerRecord containing only the fields
> needed to test record content.
>
> 2)
> public final  TestInputTopic createInputTopic(final String
> topicName, final Serde keySerde, final Serde valueSerde);
> public final  TestOutputTopic createOutputTopic(final String
> topicName, final Serde keySerde, final Serde valueSerde);
> The purpose is to create separate object for each input and output topic
> you are using. The topic name is given to createInput/OutputTopic when
> initialize topic object.
>
> For example:
>
> final TestInputTopic inputTopic1 =
> testDriver.createInputTopic( INPUT_TOPIC, longSerde, stringSerde);
> final TestInputTopic inputTopic2 =
> testDriver.createInputTopic( INPUT_TOPIC_MAP, longSerde, stringSerde);
> final TestOutputTopic outputTopic1 =
> testDriver.createOutputTopic(OUTPUT_TOPIC, longSerde, stringSerde);
> final TestOutputTopic outputTopic2 =
> testDriver.createOutputTopic(OUTPUT_TOPIC_MAP, stringSerde,
> longSerde);
> inputTopic1.pipeInput(1L, "Hello");
> assertThat(outputTopic1.readKeyValue(), equalTo(new KeyValue<>(1L,
> "Hello")));
> assertThat(outputTopic2.readKeyValue(), equalTo(new KeyValue<>("Hello",
> 1L)));
> inputTopic2.pipeInput(1L, "Hello");
>
>
> Jukka
>
> to 20. kesäk. 2019 klo 23.52 Guozhang Wang (wangg...@gmail.com) kirjoitti:
>
> > Hello Jukka,
> >
> > Thanks for writing the KIP, I have a couple of quick questions:
> >
> > 1) Is "TestRecord" an existing class that you propose to piggy-back on?
> > Right now we have a scala TestRecord case class but I doubt that was your
> > proposal, or are you proposing to add a new Java class?
> >
> > 2) Would the new API only allow a single input / output topic with
> > `createInput/OutputTopic`? If not, when we call pipeInput how to
> determine
> > which topic this record should be pipe to?
> >
> >
> > Guozhang
> >
> > On Mon, Jun 17, 2019 at 1:34 PM John Roesler  wrote:
> >
> > > Woah, I wasn't aware of that Hamcrest test style. Awesome!
> > >
> > > Thanks for the updates. I look forward to hearing what others think.
> > >
> > > -John
> > >
> > > On Mon, Jun 17, 2019 at 4:12 AM Jukka Karvanen
> > >  wrote:
> > > >
> > > > Wiki page updated:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-470%3A+TopologyTestDriver+test+input+and+output+usability+improvements
> > > >
> > > >
> > > > ClientRecord removed and replaced with TestRecord in method calls.
> > > > TestRecordFactory removed (time tracking functionality to be included
> > to
> > > > TestInputTopic)
> > > > OutputVerifier deprecated
> > > > TestRecord topic removed and getters added
> > > >
> > > > Getters in TestRecord enable writing test ignoring selected fields
> with
> > > > hamcrest like this:
> > > >
> > > > assertThat(outputTopic.readRecord(), allOf(
> > > > hasProperty("key", equalTo(1L)),
> > > > hasProperty("value", equalTo("Hello")),
> > > > hasProperty("headers", equalTo(headers;
> > > >
> > > >
> > > > Jukka
> > > >
> > > > la 15. kesäk. 2019 klo 1.10 John Roesler (j...@confluent.io)
> > kirjoitti:
> > > >
> > > > > Sounds good. Thanks as always for considering my feedback!
> > > > > -John
> > > > >
> > > > > On Fri, Jun 14, 2019 at 12:12 PM Jukka Karvanen
> > > > >  wrote:
> > > > > >
> > > > > > Ok, I will modify KIP Public Interface in a wiki based on the
> > > feedback.
> > > > > >
> > > > > > TestRecordFactory / ConsumerRecordFactory was used by
> > TestInputTopic
> > > with
> > > > > > the version I had with KIP456, but maybe I can merge That
> > > functionality
> > > > > to
> > > > > > InputTopic or  TestRecordFactory   can kept non public maybe
> moving
> > > it to
> > > > > > internals package.
> > > > > >
> > > > > > I will make the proposal with a slim down interface.
> > > > > > I don't want to go to so slim as you proposed with only
> TestRecord
> > or
> > > > > > List, because you then still end up doing helper
> > methods
> > > to
> > > > > > construct List of TestRecord.
> > > > > > The list of values is easier to write and clearer to read than if
> > you
> > > > > need
> > > > > > to contruct list of TestRecords.
> > > > > >
> > > > > > For example:
> > > > > >
> > > > > > final List inputValues = Arrays.asList(
> > > > > > "Apache Kafka Streams Example",
> > > > > > "Using Kafka Streams Test Utils",
> > > > > > "Reading and Writing Kafka Topic"
> > > > > > );
> > > > > > inputTopic.pipeValueList(inputValues);
> > > > > >
> > > > > >
> > > > > > Let's check after the next iteration is it still worth reducing
> the
> > > > > methods.
> > > > > >
> > > > > >
> > > > > > Jukka
> > > > > >
> > > > > >
> > > > > > pe 14. kesäk. 2019 klo 18.27 John Roesler 

[jira] [Resolved] (KAFKA-6408) Kafka MirrorMaker doesn't replicate messages when .* regex is used

[Waleed Fateem (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Waleed Fateem resolved KAFKA-6408.
--
Resolution: Not A Problem

> Kafka MirrorMaker doesn't replicate messages when .* regex is used
> --
>
> Key: KAFKA-6408
> URL: https://issues.apache.org/jira/browse/KAFKA-6408
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 0.11.0.0
>Reporter: Waleed Fateem
>Priority: Minor
>
> When using the regular expression .* for the whitelist parameter in Kafka 
> MirrorMaker in order to mirror all topics, the MirrorMaker doesn't replicate 
> any messages. I was then able to see messages flowing again and being 
> replicated between the two Kafka clusters once I changed the whitelist 
> configuration to use another regular expression, such as 'topic1 | topic2 | 
> topic3' 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-8580) Compute RocksDB Metrics

[Bruno Cadonna (JIRA)]

Bruno Cadonna created KAFKA-8580:


 Summary: Compute RocksDB Metrics
 Key: KAFKA-8580
 URL: https://issues.apache.org/jira/browse/KAFKA-8580
 Project: Kafka
  Issue Type: Sub-task
  Components: streams
Reporter: Bruno Cadonna


Once the metrics for RocksDB are exposed, the metrics need to be recorded in 
the RocksDB state stores.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-8579) Expose RocksDB Metrics to JMX

[Bruno Cadonna (JIRA)]

Bruno Cadonna created KAFKA-8579:


 Summary: Expose RocksDB Metrics to JMX
 Key: KAFKA-8579
 URL: https://issues.apache.org/jira/browse/KAFKA-8579
 Project: Kafka
  Issue Type: Sub-task
  Components: streams
Reporter: Bruno Cadonna


Sensors and the corresponding metrics specified to record RocksDB metrics need 
to be created in the RocksDB state stores. Once the metrics are created and 
registered in the Kafka metrics registry, they are also exposed in JMX. This 
ticker does not include the computation of the RocksDB metrics.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] 2.3.0 RC3

[Ismael Juma]

Hi Colin,

One more thing: the Maven repo was not updated, it seems. Can you fix that?

Ismael

On Fri, Jun 21, 2019 at 8:23 AM Ismael Juma  wrote:

> Hi Colin,
>
> Thanks, +1 (binding).
>
> I verified the quickstart and signatures for the source and kafka 2.12
> artifacts with Java 11. One note is that we need to complete the upgrade
> section for 2.3.0 before we announce the release. No new RC is needed, we
> just have to have it on the website.
>
> Ismael
>
> On Wed, Jun 19, 2019 at 3:03 PM Colin McCabe  wrote:
>
>> Hi all,
>>
>> We discovered some problems with the second release candidate (RC2) of
>> 2.3.0.  Specifically, KAFKA-8564.  I've created a new RC which includes the
>> fix for this issue.
>>
>> Check out the release notes for the 2.3.0 release here:
>> https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/RELEASE_NOTES.html
>>
>> The vote will go until Saturday, June 22nd, or until we create another RC.
>>
>> * Kafka's KEYS file containing PGP keys we use to sign the release can be
>> found here:
>> https://kafka.apache.org/KEYS
>>
>> * The release artifacts to be voted upon (source and binary) are here:
>> https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>
>> * Javadoc:
>> https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/javadoc/
>>
>> * The tag to be voted upon (off the 2.3 branch) is the 2.3.0 tag:
>> https://github.com/apache/kafka/releases/tag/2.3.0-rc3
>>
>> best,
>> Colin
>>
>> C.
>>
>

[jira] [Created] (KAFKA-8578) Add Sensors and Metrics to Expose RocksDB Metrics

[Bruno Cadonna (JIRA)]

Bruno Cadonna created KAFKA-8578:


 Summary: Add Sensors and Metrics to Expose RocksDB Metrics
 Key: KAFKA-8578
 URL: https://issues.apache.org/jira/browse/KAFKA-8578
 Project: Kafka
  Issue Type: Sub-task
Reporter: Bruno Cadonna


To expose RocksDB metrics as specified in KIP-471, a bunch of sensors and 
metrics in the Kafka metrics registry are required.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] 2.3.0 RC3

[Ismael Juma]

Hi Colin,

Thanks, +1 (binding).

I verified the quickstart and signatures for the source and kafka 2.12
artifacts with Java 11. One note is that we need to complete the upgrade
section for 2.3.0 before we announce the release. No new RC is needed, we
just have to have it on the website.

Ismael

On Wed, Jun 19, 2019 at 3:03 PM Colin McCabe  wrote:

> Hi all,
>
> We discovered some problems with the second release candidate (RC2) of
> 2.3.0.  Specifically, KAFKA-8564.  I've created a new RC which includes the
> fix for this issue.
>
> Check out the release notes for the 2.3.0 release here:
> https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/RELEASE_NOTES.html
>
> The vote will go until Saturday, June 22nd, or until we create another RC.
>
> * Kafka's KEYS file containing PGP keys we use to sign the release can be
> found here:
> https://kafka.apache.org/KEYS
>
> * The release artifacts to be voted upon (source and binary) are here:
> https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/javadoc/
>
> * The tag to be voted upon (off the 2.3 branch) is the 2.3.0 tag:
> https://github.com/apache/kafka/releases/tag/2.3.0-rc3
>
> best,
> Colin
>
> C.
>

Re: [VOTE] KIP-471: Expose RocksDB Metrics in Kafka Streams

[Bruno Cadonna]

Hi,

Since the voting passed the 72 hours and the number of binding votes
are reached I will close this voting.

KIP-471 passes with
- 3 binding votes (Bill, Guozhang, Matthias)
- 2 non-binding votes (John, Patrik)

Thank you to all for the discussion and the voting.

Best,
Bruno

On Thu, Jun 20, 2019 at 11:14 PM Matthias J. Sax  wrote:
>
> +1 (binding)
>
>
> On 6/20/19 11:53 AM, Guozhang Wang wrote:
> > +1 (binding)
> >
> > Thanks Bruno!
> >
> > Would also be interested to see how much overhead it may incur by enabling
> > DEBUG metrics now, if it is huge we may consider doing finer-grained
> > metrics enabling, but that would be another follow-up task.
> >
> > Guozhang
> >
> > On Wed, Jun 19, 2019 at 1:37 PM Patrik Kleindl  wrote:
> >
> >> +1 (non-binding)
> >> Thanks!
> >> Best regards
> >> Patrik
> >>
> >>> Am 19.06.2019 um 21:55 schrieb Bill Bejeck :
> >>>
> >>> +1 (binding)
> >>>
> >>> Thanks,
> >>> Bill
> >>>
>  On Wed, Jun 19, 2019 at 1:19 PM John Roesler  wrote:
> 
>  I'm +1 (nonbinding)
> 
>  Thanks!
>  -John
> 
> > On Tue, Jun 18, 2019 at 7:48 AM Bruno Cadonna 
> >> wrote:
> >
> > Hi,
> >
> > I would like to start the voting on KIP-471:
> >
> 
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-471%3A+Expose+RocksDB+Metrics+in+Kafka+Streams
> >
> > You can find the discussion here:
> >
> 
> >> https://lists.apache.org/thread.html/125bdd984fe0667962018da6ce10bce6d5895c5103955a8e4c730fef@%3Cdev.kafka.apache.org%3E
> >
> > Best,
> > Bruno
> 
> >>
> >
> >
>

Re: [VOTE] KIP-476: Add Java AdminClient interface

[Ismael Juma]

This is even more reason not to deprecate immediately, there is very little
maintenance cost for us. We should be mindful that many of our users (eg
Spark, Flink, etc.) typically allow users to specify the kafka clients
version and hence avoid using new classes/interfaces for some time. They
would get a bunch of warnings they cannot do anything about apart from
suppressing.

Ismael

On Fri, Jun 21, 2019 at 4:00 AM Andy Coates  wrote:

> Hi Ismael,
>
> I’m happy enough to not deprecate the existing `AdminClient` class as part
> of this change.
>
> However, note that, the class will likely be empty, i.e. all methods and
> implementations will be inherited from the interface:
>
> public abstract class AdminClient implements Admin {
> }
>
> Not marking it as deprecated has the benefit that users won’t see any
> deprecation warnings on the next release. Conversely, deprecating it will
> mean we can choose to remove this, now pointless class, in the future if we
> choose.
>
> That’s my thinking for deprecation, but as I’ve said I’m happy either way.
>
> Andy
>
> > On 18 Jun 2019, at 16:09, Ismael Juma  wrote:
> >
> > I agree with Ryanne, I think we should avoid deprecating AdminClient and
> > causing so much churn for users who don't actually care about this niche
> > use case.
> >
> > Ismael
> >
> > On Tue, Jun 18, 2019 at 6:43 AM Andy Coates  wrote:
> >
> >> Hi Ryanne,
> >>
> >> If we don't change the client code, then everywhere will still expect
> >> subclasses of `AdminClient`, so the interface will be of no use, i.e. I
> >> can't write a class that implements the new interface and pass it to the
> >> client code.
> >>
> >> Thanks,
> >>
> >> Andy
> >>
> >> On Mon, 17 Jun 2019 at 19:01, Ryanne Dolan 
> wrote:
> >>
> >>> Andy, while I agree that the new interface is useful, I'm not convinced
> >>> adding an interface requires deprecating AdminClient and changing so
> much
> >>> client code. Why not just add the Admin interface, have AdminClient
> >>> implement it, and have done?
> >>>
> >>> Ryanne
> >>>
> >>> On Mon, Jun 17, 2019 at 12:09 PM Andy Coates 
> wrote:
> >>>
>  Hi all,
> 
>  I think I've addressed all concerns. Let me know if I've not.  Can I
> >> call
>  another round of votes please?
> 
>  Thanks,
> 
>  Andy
> 
>  On Fri, 14 Jun 2019 at 04:55, Satish Duggana <
> satish.dugg...@gmail.com
> >>>
>  wrote:
> 
> > Hi Andy,
> > Thanks for the KIP. This is a good change and it gives the user a
> >>> better
> > handle on Admin client usage. I agree with the proposal except the
> >> new
> > `Admin` interface having all the methods from `AdminClient` abstract
>  class.
> > It should be kept clean having only the admin operations as methods
> >>> from
> > KafkaClient abstract class but not the factory methods as mentioned
> >> in
>  the
> > earlier mail.
> >
> > I know about dynamic proxies(which were widely used in RMI/EJB
> >> world).
> >>> I
>  am
> > curious about the usecase using dynamic proxies with Admin client
> > interface. Dynamic proxy can have performance penalty if it is used
> >> in
> > critical path. Is that the primary motivation for creating the KIP?
> >
> > Thanks,
> > Satish.
> >
> > On Wed, Jun 12, 2019 at 8:43 PM Andy Coates 
> >> wrote:
> >
> >> I'm not married to that part.  That was only done to keep it more
> >> or
>  less
> >> inline with what's already there, (an abstract class that has a
> >>> factory
> >> method that returns a subclass sounds like the same
> >> anti-pattern
>  ;))
> >>
> >> An alternative would to have an `AdminClients` utility class to
> >>> create
> > the
> >> admin client.
> >>
> >> On Mon, 10 Jun 2019 at 19:31, Matthias J. Sax <
> >> matth...@confluent.io
> 
> >> wrote:
> >>
> >>> Hmmm...
> >>>
> >>> So the new interface, returns an instance of a class that
> >>> implements
> > the
> >>> interface. This sounds a little bit like an anti-pattern?
> >> Shouldn't
> >>> interfaces actually not know anything about classes that
> >> implement
>  the
> >>> interface?
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 6/10/19 11:22 AM, Andy Coates wrote:
>  `AdminClient` would be deprecated purely because it would no
> >>> longer
> >> serve
>  any purpose and would be virtually empty, getting all of its
> >>> implementation
>  from the new interfar. It would be nice to remove this from the
> >>> API
> > at
> >>> the
>  next major version bump, hence the need to deprecate.
> 
>  `AdminClient.create()` would return what it does today, (so
> >> not a
> >>> breaking
>  change).
> 
>  On Tue, 4 Jun 2019 at 22:24, Ryanne Dolan <
> >> ryannedo...@gmail.com
> 
> >> wrote:
> 
> >> The existing `AdminClient` will be marked as 

Re: [VOTE] KIP-476: Add Java AdminClient interface

[Ismael Juma]

Hi Andy,

I didn't see any reason being mentioned why it's an anti pattern or cleaner
so I assume it's subjective. :) For what it's worth, the approach of a
collection interface providing a default implementation is very common in
Scala and it makes a lot of sense in my mind. For example, why does
everyone have to decide the List implementation, ArrayList is the best
default and it should just be chosen for the common case.

Having a separate class with the same name and a suffix at the end just
adds complexity and surface area while making it less discoverable for very
little benefit in my opinion.

Ismael

On Fri, Jun 21, 2019 at 3:56 AM Andy Coates  wrote:

> Hi Ismael,
>
> Matthias thought having the interface also provide a factory method that
> returns a specific implementation was a bit of an anti-pattern, and I would
> tend to agree, though I’ve used this same pattern myself at times where the
> set of implementations is known.
>
> Matthias may want to answer with his own thoughts, but from my own
> experience I’ve often found it cleaner to have a companion utility class to
> an interface in an API, that has factory methods to create instances of the
> common set of implementations. In this case it would be `KafkaAdminClient`
> only at the moment, though if we were to support something like the
> `DelegatingAdminClient` that Colin mentioned in the future, this too could
> be on the utility class.
>
> My preference is the separate classes. But in the interest of moving this
> forward I’m happy with either.
>
> Andy
>
>
>
> > On 18 Jun 2019, at 16:07, Ismael Juma  wrote:
> >
> > I don't agree with this change. The idea that an interface cannot have a
> > default implementation is outdated in my view. Can someone provide any
> > benefit to introducing a separate class for the factory method?
> >
> > Ismael
> >
> > On Mon, Jun 17, 2019 at 10:07 AM Andy Coates  wrote:
> >
> >> Hi All,
> >>
> >> I've updated the KIP to move the `create` factory method implementation
> >> into a new `AdminClients` utility class, rather than on the new `Admin`
> >> interface.
> >>
> >> Satish,
> >>
> >> As above, the KIP has been updated to only have the operations on the
> >> `Admin` api. As for the overhead of dynamic proxies... the use of
> dynamic
> >> proxies is totally up to the users of the library. In KSQL we use
> dynamic
> >> proxies because the overhead is acceptable and it decouples us from
> >> additions to the client interfaces. Others may decide otherwise for
> their
> >> project. By making the admin api an interface we're empowering users to
> >> choose the right approach for them.
> >>
> >> This is the primary motivation for the KIP from my point of view.
> However,
> >> it also brings it inline with the other Kafka clients, and gives users
> the
> >> freedom to do what they want, rather than requiring the use of an
> abstract
> >> base class.
> >>
> >> Thanks,
> >>
> >> Andy
> >>
> >>
> >> On Fri, 14 Jun 2019 at 04:55, Satish Duggana 
> >> wrote:
> >>
> >>> Hi Andy,
> >>> Thanks for the KIP. This is a good change and it gives the user a
> better
> >>> handle on Admin client usage. I agree with the proposal except the new
> >>> `Admin` interface having all the methods from `AdminClient` abstract
> >> class.
> >>> It should be kept clean having only the admin operations as methods
> from
> >>> KafkaClient abstract class but not the factory methods as mentioned in
> >> the
> >>> earlier mail.
> >>>
> >>> I know about dynamic proxies(which were widely used in RMI/EJB world).
> I
> >> am
> >>> curious about the usecase using dynamic proxies with Admin client
> >>> interface. Dynamic proxy can have performance penalty if it is used in
> >>> critical path. Is that the primary motivation for creating the KIP?
> >>>
> >>> Thanks,
> >>> Satish.
> >>>
> >>> On Wed, Jun 12, 2019 at 8:43 PM Andy Coates  wrote:
> >>>
>  I'm not married to that part.  That was only done to keep it more or
> >> less
>  inline with what's already there, (an abstract class that has a
> factory
>  method that returns a subclass sounds like the same anti-pattern
> >> ;))
> 
>  An alternative would to have an `AdminClients` utility class to create
> >>> the
>  admin client.
> 
>  On Mon, 10 Jun 2019 at 19:31, Matthias J. Sax 
>  wrote:
> 
> > Hmmm...
> >
> > So the new interface, returns an instance of a class that implements
> >>> the
> > interface. This sounds a little bit like an anti-pattern? Shouldn't
> > interfaces actually not know anything about classes that implement
> >> the
> > interface?
> >
> >
> > -Matthias
> >
> > On 6/10/19 11:22 AM, Andy Coates wrote:
> >> `AdminClient` would be deprecated purely because it would no longer
>  serve
> >> any purpose and would be virtually empty, getting all of its
> > implementation
> >> from the new interfar. It would be nice to remove this from the API
> >>> at
> > the
> >> 

Re: [VOTE] 2.3.0 RC3

[Kamal Chandraprakash]

+1 (non-binding)

* Ran unit and integration test on 2.11 and 2.12
* Verified quick start
* Ran internal apps on the 3 node cluster

On Thu, Jun 20, 2019 at 3:33 AM Colin McCabe  wrote:

> Hi all,
>
> We discovered some problems with the second release candidate (RC2) of
> 2.3.0.  Specifically, KAFKA-8564.  I've created a new RC which includes the
> fix for this issue.
>
> Check out the release notes for the 2.3.0 release here:
> https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/RELEASE_NOTES.html
>
> The vote will go until Saturday, June 22nd, or until we create another RC.
>
> * Kafka's KEYS file containing PGP keys we use to sign the release can be
> found here:
> https://kafka.apache.org/KEYS
>
> * The release artifacts to be voted upon (source and binary) are here:
> https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~cmccabe/kafka-2.3.0-rc3/javadoc/
>
> * The tag to be voted upon (off the 2.3 branch) is the 2.3.0 tag:
> https://github.com/apache/kafka/releases/tag/2.3.0-rc3
>
> best,
> Colin
>
> C.
>

Possible implementation for KAFKA-560

[Carlos Manuel Duclos-Vergara]

Hi,

This is an ancient task, but I feel it is still current today (specially
since as somebody that deals with a Kafka cluster I know that this happens
more often than not).

The task is about garbage collection of topics in a sort of automated way.
After some consideration I started a prototype implementation based on a
manual process:

1. Using the cli, I can use the --describe-topic to get a list of topics
that have size 0
2. Massage that list into something that can be then fed into the cli and
remove the topics that have size 0.

The guiding principle here is the assumption that abandoned topics will
eventually have size 0, because all records will expire. This is not true
for all topics, but it covers a large portion of them and having something
like this would help admins to find "suspicious" topics at least.

I started implementing this change and I realized that it would require a
change in the protocol, because the sizes are never sent over the wire.
Funny enough we collect the sizes of the log files, but we do not send them.

I think this kind of changes will require a KIP, but I wanted to ask what
others think about this.

The in-progress implementation of this can be found here:
https://github.com/carlosduclos/kafka/commit/0dffe5e131c3bd32b77f56b9be8eded89a96df54

Comments?

-- 
Carlos Manuel Duclos Vergara
Backend Software Developer

Re: [VOTE] KIP-476: Add Java AdminClient interface

[Andy Coates]

Hi Ismael,

I’m happy enough to not deprecate the existing `AdminClient` class as part of 
this change.

However, note that, the class will likely be empty, i.e. all methods and 
implementations will be inherited from the interface:

public abstract class AdminClient implements Admin {
}

Not marking it as deprecated has the benefit that users won’t see any 
deprecation warnings on the next release. Conversely, deprecating it will mean 
we can choose to remove this, now pointless class, in the future if we choose. 

That’s my thinking for deprecation, but as I’ve said I’m happy either way.

Andy

> On 18 Jun 2019, at 16:09, Ismael Juma  wrote:
> 
> I agree with Ryanne, I think we should avoid deprecating AdminClient and
> causing so much churn for users who don't actually care about this niche
> use case.
> 
> Ismael
> 
> On Tue, Jun 18, 2019 at 6:43 AM Andy Coates  wrote:
> 
>> Hi Ryanne,
>> 
>> If we don't change the client code, then everywhere will still expect
>> subclasses of `AdminClient`, so the interface will be of no use, i.e. I
>> can't write a class that implements the new interface and pass it to the
>> client code.
>> 
>> Thanks,
>> 
>> Andy
>> 
>> On Mon, 17 Jun 2019 at 19:01, Ryanne Dolan  wrote:
>> 
>>> Andy, while I agree that the new interface is useful, I'm not convinced
>>> adding an interface requires deprecating AdminClient and changing so much
>>> client code. Why not just add the Admin interface, have AdminClient
>>> implement it, and have done?
>>> 
>>> Ryanne
>>> 
>>> On Mon, Jun 17, 2019 at 12:09 PM Andy Coates  wrote:
>>> 
 Hi all,
 
 I think I've addressed all concerns. Let me know if I've not.  Can I
>> call
 another round of votes please?
 
 Thanks,
 
 Andy
 
 On Fri, 14 Jun 2019 at 04:55, Satish Duggana >> 
 wrote:
 
> Hi Andy,
> Thanks for the KIP. This is a good change and it gives the user a
>>> better
> handle on Admin client usage. I agree with the proposal except the
>> new
> `Admin` interface having all the methods from `AdminClient` abstract
 class.
> It should be kept clean having only the admin operations as methods
>>> from
> KafkaClient abstract class but not the factory methods as mentioned
>> in
 the
> earlier mail.
> 
> I know about dynamic proxies(which were widely used in RMI/EJB
>> world).
>>> I
 am
> curious about the usecase using dynamic proxies with Admin client
> interface. Dynamic proxy can have performance penalty if it is used
>> in
> critical path. Is that the primary motivation for creating the KIP?
> 
> Thanks,
> Satish.
> 
> On Wed, Jun 12, 2019 at 8:43 PM Andy Coates 
>> wrote:
> 
>> I'm not married to that part.  That was only done to keep it more
>> or
 less
>> inline with what's already there, (an abstract class that has a
>>> factory
>> method that returns a subclass sounds like the same
>> anti-pattern
 ;))
>> 
>> An alternative would to have an `AdminClients` utility class to
>>> create
> the
>> admin client.
>> 
>> On Mon, 10 Jun 2019 at 19:31, Matthias J. Sax <
>> matth...@confluent.io
 
>> wrote:
>> 
>>> Hmmm...
>>> 
>>> So the new interface, returns an instance of a class that
>>> implements
> the
>>> interface. This sounds a little bit like an anti-pattern?
>> Shouldn't
>>> interfaces actually not know anything about classes that
>> implement
 the
>>> interface?
>>> 
>>> 
>>> -Matthias
>>> 
>>> On 6/10/19 11:22 AM, Andy Coates wrote:
 `AdminClient` would be deprecated purely because it would no
>>> longer
>> serve
 any purpose and would be virtually empty, getting all of its
>>> implementation
 from the new interfar. It would be nice to remove this from the
>>> API
> at
>>> the
 next major version bump, hence the need to deprecate.
 
 `AdminClient.create()` would return what it does today, (so
>> not a
>>> breaking
 change).
 
 On Tue, 4 Jun 2019 at 22:24, Ryanne Dolan <
>> ryannedo...@gmail.com
 
>> wrote:
 
>> The existing `AdminClient` will be marked as deprecated.
> 
> What's the reasoning behind this? I'm fine with the other
>>> changes,
> but
> would prefer to keep the existing public API intact if it's
>> not
>> hurting
> anything.
> 
> Also, what will AdminClient.create() return? Would it be a
 breaking
>>> change?
> 
> Ryanne
> 
> On Tue, Jun 4, 2019, 11:17 AM Andy Coates 
> wrote:
> 
>> Hi folks
>> 
>> As there's been no chatter on this KIP I'm assuming it's
>>> non-contentious,
>> (or just boring), hence I'd like to call a vote for KIP-476:
>> 
>> 
>> 
>>> 
>> 
> 
 
>>> 
>> 

Re: [VOTE] KIP-476: Add Java AdminClient interface

[Andy Coates]

Hi Ismael,

Matthias thought having the interface also provide a factory method that 
returns a specific implementation was a bit of an anti-pattern, and I would 
tend to agree, though I’ve used this same pattern myself at times where the set 
of implementations is known.

Matthias may want to answer with his own thoughts, but from my own experience 
I’ve often found it cleaner to have a companion utility class to an interface 
in an API, that has factory methods to create instances of the common set of 
implementations. In this case it would be `KafkaAdminClient` only at the 
moment, though if we were to support something like the `DelegatingAdminClient` 
that Colin mentioned in the future, this too could be on the utility class.

My preference is the separate classes. But in the interest of moving this 
forward I’m happy with either.

Andy



> On 18 Jun 2019, at 16:07, Ismael Juma  wrote:
> 
> I don't agree with this change. The idea that an interface cannot have a
> default implementation is outdated in my view. Can someone provide any
> benefit to introducing a separate class for the factory method?
> 
> Ismael
> 
> On Mon, Jun 17, 2019 at 10:07 AM Andy Coates  wrote:
> 
>> Hi All,
>> 
>> I've updated the KIP to move the `create` factory method implementation
>> into a new `AdminClients` utility class, rather than on the new `Admin`
>> interface.
>> 
>> Satish,
>> 
>> As above, the KIP has been updated to only have the operations on the
>> `Admin` api. As for the overhead of dynamic proxies... the use of dynamic
>> proxies is totally up to the users of the library. In KSQL we use dynamic
>> proxies because the overhead is acceptable and it decouples us from
>> additions to the client interfaces. Others may decide otherwise for their
>> project. By making the admin api an interface we're empowering users to
>> choose the right approach for them.
>> 
>> This is the primary motivation for the KIP from my point of view. However,
>> it also brings it inline with the other Kafka clients, and gives users the
>> freedom to do what they want, rather than requiring the use of an abstract
>> base class.
>> 
>> Thanks,
>> 
>> Andy
>> 
>> 
>> On Fri, 14 Jun 2019 at 04:55, Satish Duggana 
>> wrote:
>> 
>>> Hi Andy,
>>> Thanks for the KIP. This is a good change and it gives the user a better
>>> handle on Admin client usage. I agree with the proposal except the new
>>> `Admin` interface having all the methods from `AdminClient` abstract
>> class.
>>> It should be kept clean having only the admin operations as methods from
>>> KafkaClient abstract class but not the factory methods as mentioned in
>> the
>>> earlier mail.
>>> 
>>> I know about dynamic proxies(which were widely used in RMI/EJB world). I
>> am
>>> curious about the usecase using dynamic proxies with Admin client
>>> interface. Dynamic proxy can have performance penalty if it is used in
>>> critical path. Is that the primary motivation for creating the KIP?
>>> 
>>> Thanks,
>>> Satish.
>>> 
>>> On Wed, Jun 12, 2019 at 8:43 PM Andy Coates  wrote:
>>> 
 I'm not married to that part.  That was only done to keep it more or
>> less
 inline with what's already there, (an abstract class that has a factory
 method that returns a subclass sounds like the same anti-pattern
>> ;))
 
 An alternative would to have an `AdminClients` utility class to create
>>> the
 admin client.
 
 On Mon, 10 Jun 2019 at 19:31, Matthias J. Sax 
 wrote:
 
> Hmmm...
> 
> So the new interface, returns an instance of a class that implements
>>> the
> interface. This sounds a little bit like an anti-pattern? Shouldn't
> interfaces actually not know anything about classes that implement
>> the
> interface?
> 
> 
> -Matthias
> 
> On 6/10/19 11:22 AM, Andy Coates wrote:
>> `AdminClient` would be deprecated purely because it would no longer
 serve
>> any purpose and would be virtually empty, getting all of its
> implementation
>> from the new interfar. It would be nice to remove this from the API
>>> at
> the
>> next major version bump, hence the need to deprecate.
>> 
>> `AdminClient.create()` would return what it does today, (so not a
> breaking
>> change).
>> 
>> On Tue, 4 Jun 2019 at 22:24, Ryanne Dolan 
 wrote:
>> 
 The existing `AdminClient` will be marked as deprecated.
>>> 
>>> What's the reasoning behind this? I'm fine with the other changes,
>>> but
>>> would prefer to keep the existing public API intact if it's not
 hurting
>>> anything.
>>> 
>>> Also, what will AdminClient.create() return? Would it be a
>> breaking
> change?
>>> 
>>> Ryanne
>>> 
>>> On Tue, Jun 4, 2019, 11:17 AM Andy Coates 
>>> wrote:
>>> 
 Hi folks
 
 As there's been no chatter on this KIP I'm assuming it's
> non-contentious,
 (or just boring), hence I'd like to call a 

Build failed in Jenkins: kafka-trunk-jdk11 #648

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Remove legacy kafka.admin.AdminClient (#6947)

--
[...truncated 2.43 MB...]
org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestampWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareKeyValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareKeyValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 

Build failed in Jenkins: kafka-trunk-jdk8 #3740

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Remove legacy kafka.admin.AdminClient (#6947)

--
[...truncated 2.52 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

[jira] [Created] (KAFKA-8577) Flaky Test `DistributedHerderTest.testJoinLeaderCatchUpFails`

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-8577:
--

 Summary: Flaky Test 
`DistributedHerderTest.testJoinLeaderCatchUpFails`
 Key: KAFKA-8577
 URL: https://issues.apache.org/jira/browse/KAFKA-8577
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson


Started seeing this regularly:
{code:java}
java.lang.AssertionError: 
  Unexpected method call WorkerGroupMember.maybeLeaveGroup("taking too long to 
read the log"):
WorkerGroupMember.ensureActive(): expected: 2, actual: 1
WorkerGroupMember.wakeup(): expected: 2, actual: 1
WorkerGroupMember.maybeLeaveGroup("test join leader catch up fails"): 
expected: 1, actual: 0
WorkerGroupMember.requestRejoin(): expected: 1, actual: 0
WorkerGroupMember.poll(): expected: 1, actual: 0{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-470: TopologyTestDriver test input and output usability improvements

[Jukka Karvanen]

Hi  Guozhang,

1) This TestRecord is new class in my proposal. So it is a simplified
version of ProducerRecord and ConsumerRecord containing only the fields
needed to test record content.

2)
public final  TestInputTopic createInputTopic(final String
topicName, final Serde keySerde, final Serde valueSerde);
public final  TestOutputTopic createOutputTopic(final String
topicName, final Serde keySerde, final Serde valueSerde);
The purpose is to create separate object for each input and output topic
you are using. The topic name is given to createInput/OutputTopic when
initialize topic object.

For example:

final TestInputTopic inputTopic1 =
testDriver.createInputTopic( INPUT_TOPIC, longSerde, stringSerde);
final TestInputTopic inputTopic2 =
testDriver.createInputTopic( INPUT_TOPIC_MAP, longSerde, stringSerde);
final TestOutputTopic outputTopic1 =
testDriver.createOutputTopic(OUTPUT_TOPIC, longSerde, stringSerde);
final TestOutputTopic outputTopic2 =
testDriver.createOutputTopic(OUTPUT_TOPIC_MAP, stringSerde,
longSerde);
inputTopic1.pipeInput(1L, "Hello");
assertThat(outputTopic1.readKeyValue(), equalTo(new KeyValue<>(1L, "Hello")));
assertThat(outputTopic2.readKeyValue(), equalTo(new KeyValue<>("Hello", 1L)));
inputTopic2.pipeInput(1L, "Hello");


Jukka

to 20. kesäk. 2019 klo 23.52 Guozhang Wang (wangg...@gmail.com) kirjoitti:

> Hello Jukka,
>
> Thanks for writing the KIP, I have a couple of quick questions:
>
> 1) Is "TestRecord" an existing class that you propose to piggy-back on?
> Right now we have a scala TestRecord case class but I doubt that was your
> proposal, or are you proposing to add a new Java class?
>
> 2) Would the new API only allow a single input / output topic with
> `createInput/OutputTopic`? If not, when we call pipeInput how to determine
> which topic this record should be pipe to?
>
>
> Guozhang
>
> On Mon, Jun 17, 2019 at 1:34 PM John Roesler  wrote:
>
> > Woah, I wasn't aware of that Hamcrest test style. Awesome!
> >
> > Thanks for the updates. I look forward to hearing what others think.
> >
> > -John
> >
> > On Mon, Jun 17, 2019 at 4:12 AM Jukka Karvanen
> >  wrote:
> > >
> > > Wiki page updated:
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-470%3A+TopologyTestDriver+test+input+and+output+usability+improvements
> > >
> > >
> > > ClientRecord removed and replaced with TestRecord in method calls.
> > > TestRecordFactory removed (time tracking functionality to be included
> to
> > > TestInputTopic)
> > > OutputVerifier deprecated
> > > TestRecord topic removed and getters added
> > >
> > > Getters in TestRecord enable writing test ignoring selected fields with
> > > hamcrest like this:
> > >
> > > assertThat(outputTopic.readRecord(), allOf(
> > > hasProperty("key", equalTo(1L)),
> > > hasProperty("value", equalTo("Hello")),
> > > hasProperty("headers", equalTo(headers;
> > >
> > >
> > > Jukka
> > >
> > > la 15. kesäk. 2019 klo 1.10 John Roesler (j...@confluent.io)
> kirjoitti:
> > >
> > > > Sounds good. Thanks as always for considering my feedback!
> > > > -John
> > > >
> > > > On Fri, Jun 14, 2019 at 12:12 PM Jukka Karvanen
> > > >  wrote:
> > > > >
> > > > > Ok, I will modify KIP Public Interface in a wiki based on the
> > feedback.
> > > > >
> > > > > TestRecordFactory / ConsumerRecordFactory was used by
> TestInputTopic
> > with
> > > > > the version I had with KIP456, but maybe I can merge That
> > functionality
> > > > to
> > > > > InputTopic or  TestRecordFactory   can kept non public maybe moving
> > it to
> > > > > internals package.
> > > > >
> > > > > I will make the proposal with a slim down interface.
> > > > > I don't want to go to so slim as you proposed with only TestRecord
> or
> > > > > List, because you then still end up doing helper
> methods
> > to
> > > > > construct List of TestRecord.
> > > > > The list of values is easier to write and clearer to read than if
> you
> > > > need
> > > > > to contruct list of TestRecords.
> > > > >
> > > > > For example:
> > > > >
> > > > > final List inputValues = Arrays.asList(
> > > > > "Apache Kafka Streams Example",
> > > > > "Using Kafka Streams Test Utils",
> > > > > "Reading and Writing Kafka Topic"
> > > > > );
> > > > > inputTopic.pipeValueList(inputValues);
> > > > >
> > > > >
> > > > > Let's check after the next iteration is it still worth reducing the
> > > > methods.
> > > > >
> > > > >
> > > > > Jukka
> > > > >
> > > > >
> > > > > pe 14. kesäk. 2019 klo 18.27 John Roesler (j...@confluent.io)
> > kirjoitti:
> > > > >
> > > > > > Thanks, Jukka,
> > > > > >
> > > > > > Ok, I buy this reasoning.
> > > > > >
> > > > > > Just to echo what I think I read, you would just drop
> ClientRecord
> > > > > > from the proposal, and TestRecord would stand on its own, with
> the
> > > > > > same methods and properties you proposed, and the "input topic"
> > would
> > > > > > accept TestRecord, and the "output topic" would produce
> TestRecord?
> > > > > >