Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

2019-08-20 Thread John Roesler
er you to be willing to pay another >> rebalance, >> > >> along >> > >> > > with potentially more restoration lag", and the current >> definition >> > of >> > >> > > rebalance_factor can be considered as a ro

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

2019-08-20 Thread John Roesler
ne can argue that a finer grained > measurement > > >> could > > >> > > be "resource footprint" like CPU / storage of each instance like > we > > >> have > > >> > in > > >> > > Kafka broker auto balancing tool

[jira] [Created] (KAFKA-8770) Either switch to or add an option for emit-on-change

2019-08-08 Thread John Roesler (JIRA)
John Roesler created KAFKA-8770: --- Summary: Either switch to or add an option for emit-on-change Key: KAFKA-8770 URL: https://issues.apache.org/jira/browse/KAFKA-8770 Project: Kafka Issue Type

[jira] [Created] (KAFKA-8769) Consider computing stream time independently per key

2019-08-08 Thread John Roesler (JIRA)
John Roesler created KAFKA-8769: --- Summary: Consider computing stream time independently per key Key: KAFKA-8769 URL: https://issues.apache.org/jira/browse/KAFKA-8769 Project: Kafka Issue Type

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

2019-08-07 Thread John Roesler
Hey Guozhang, Thanks for the review! 1. Yes, even with `num.standby.replicas := 0`, we will still temporarily allocate standby tasks to accomplish a no-downtime task migration. Although, I'd argue that this doesn't really violate the config, as the task isn't a true hot standby. As soon as it cat

Re: [VOTE] KIP-478 Strongly Typed Streams Processor API

2019-07-30 Thread John Roesler
ng Wang > wrote: > > > >> Yeah I think I agree with you. > >> > >> +1 (binding) from me. > >> > >> > >> Guozhang > >> > >> > >> On Wed, Jul 24, 2019 at 7:43 AM John Roesler wrote: > >> > >>> Hi Gu

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

2019-07-30 Thread John Roesler
rade path if we > change the `ProcessorContext` in the future. > > Thus, I think we can move forward. > > > -Matthias > > On 7/24/19 3:32 PM, John Roesler wrote: > > Hey again Matthias, > > > > I think it might help to frame the evaluation of the Context qu

Re: [DISCUSS] KIP-470: TopologyTestDriver test input and output usability improvements

2019-07-30 Thread John Roesler
and this may be me being paranoid, > but > > >>>> will > > >>>>> the changes for input/output topic affect how the > TopologyTestDriver > > >>>> works > > >>>>> with internal topics when there are

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

2019-07-24 Thread John Roesler
texts, likewise with DeserializationExceptionHandler and StateStore. In other words, I _think_ that we have a clean migration path to address the Context problem in follow-on work. But clearly my motivation may be corrupt. What do you think? Thanks, -John On Wed, Jul 24, 2019 at 5:06 PM John Ro

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

2019-07-24 Thread John Roesler
path will > smooth, we can of course do a follow up KIP. Another possibility would > be, to still do an extra KIP but ensure that both KIPs are contained in > the same release. > > WDYT? > > > -Matthias > > On 7/24/19 11:55 AM, John Roesler wrote: > > Hey Matthias, &

Re: [VOTE] KIP-466: Add support for List serialization and deserialization

2019-07-24 Thread John Roesler
Thanks, Daniyar, I'm +1 (nonbinding) -John On Tue, Jun 11, 2019 at 1:45 PM Development wrote: > Hello, > > I want to start a vote for KIP-466 < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-466:+Add+support+for+List%3CT%3E+serialization+and+deserialization > < > https://cwiki.apache.

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

2019-07-24 Thread John Roesler
sor("p1", new ProcessorSupplier BarValue>()..., "s"); > > t.addProcessor("p2", new ProcessorSupplier VOut>()..., "p1"); > > Just want to make sure users understand the impact/scope of the change, > especially what is _not_ achieved.

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

2019-07-24 Thread John Roesler
; `StreamsConfig#DEFAULT_WINDOWED_KEY_SERDE_INNER_CLASS`. > > > > > Btw: I just realized that we actually don't need `ProducerConfig` > > > list.key/value.serializer.type > > because the list-type is irrelevant on write. We only need `inner` config. > > &

Re: [VOTE] KIP-478 Strongly Typed Streams Processor API

2019-07-24 Thread John Roesler
question regarding Transformer in the DISCUSS thread. Other > than that I think this KIP is ready. Thanks! > > > Guozhang > > > On Tue, Jul 16, 2019 at 9:01 AM John Roesler wrote: > > > Hi Dev, > > > > After a good discussion, I'd like to start the

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

2019-07-24 Thread John Roesler
upplier where we allow at most one > `context.forward()` AND we ignore whatever passed in as key but just use > the original key. > > > Guozhang > > > On Tue, Jul 16, 2019 at 9:03 AM John Roesler wrote: > > > Hi again, all, > > > > I have started th

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

2019-07-23 Thread John Roesler
efault.list.key/value.serde.type > >>>> default.list.key/value.serde.inner > >>>> > >>>> > >>>> Adding `d.l.k/v.serde.t/i` to `CommonClientConfigs does not sound > right > >>>> to me. Also note, that it seems better to avoid the

[jira] [Created] (KAFKA-8696) Clean up Sum/Count/Total metrics

2019-07-22 Thread John Roesler (JIRA)
John Roesler created KAFKA-8696: --- Summary: Clean up Sum/Count/Total metrics Key: KAFKA-8696 URL: https://issues.apache.org/jira/browse/KAFKA-8696 Project: Kafka Issue Type: Improvement

Re: [VOTE] KIP-488: Clean up Sum,Count,Total Metrics

2019-07-22 Thread John Roesler
thias J. Sax wrote: > > +1 (binding) > > > On 7/17/19 1:20 PM, Bill Bejeck wrote: > > +1 (binding) for the updated KIP. > > > > On Wed, Jul 17, 2019 at 4:09 PM John Roesler wrote: > > > >> Hey, Bruno and Bill, > >> > >> Since you

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

2019-07-17 Thread John Roesler
ani > > > On Jul 17, 2019, at 11:11 PM, John Roesler wrote: > > > > Hey, all, just to chime in, > > > > I think it might be useful to have an option to specify the > > partitioner. The case I have in mind is that some data may get > > repartitioned a

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

2019-07-17 Thread John Roesler
Hey, all, just to chime in, I think it might be useful to have an option to specify the partitioner. The case I have in mind is that some data may get repartitioned and then joined with an input topic. If the right-side input topic uses a custom partitioning strategy, then the repartitioned stream

Re: [VOTE] KIP-488: Clean up Sum,Count,Total Metrics

2019-07-17 Thread John Roesler
e roughly synonymous. I'm already > > > scratching my head at what "TotalSum" means. It's clear in the context of > > > your matrix, juxtaposed with the alternatives, but when I come across the > > > name in isolation I suspect I'll be back looking

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

2019-07-17 Thread John Roesler
Cool. I'll update the KIP, then, and we can re-start the vote. (looks like you've already cast a vote :) ) -John On Wed, Jul 17, 2019 at 1:23 PM Ryanne Dolan wrote: > > John, makes sense to me! Thanks. > > Ryanne > > On Wed, Jul 17, 2019, 1:16 PM John Roesler wrot

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

2019-07-17 Thread John Roesler
arly that "Sum" is a "sum of values" whereas > "Count" is a "number of things" -- but that doesn't need to be part of the > KIP > > On Wed, Jul 17, 2019 at 11:00 AM John Roesler wrote: > > > Thanks for the replies. > > >

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

2019-07-17 Thread John Roesler
vanced' or non-simple kind of Windowed?" than they are to > wonder what is the weighting behind these metrics. > > Sophie > > On Wed, Jul 17, 2019 at 8:18 AM John Roesler wrote: > > > Thanks for the discussion, all. > > > > I've done a little more r

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

2019-07-17 Thread John Roesler
Windowed, was about to suggest that as I was catching up on the > > discussion but Bill beat me to it :) > > > > On Tue, Jul 16, 2019 at 2:23 PM Bill Bejeck wrote: > > > >> Hi John, > >> > >> Thanks for the updates. > >> > >> I like Runn

Re: [VOTE] KIP-466: Add support for List serialization and deserialization

2019-07-16 Thread John Roesler
Thanks, Daniyar! I'm +1 (nonbinding) -John On Tue, Jul 16, 2019 at 3:14 PM Development wrote: > > Hi, > > I’d like to start a vote thread for KIP-466. > > This addition will introduce new serde type ListSerde. > > More info at: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-466%3A+Add

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

2019-07-16 Thread John Roesler
Count, RunningSum (Instead of Total) Thanks, -John On Tue, Jul 16, 2019 at 3:04 PM Ryanne Dolan wrote: > > John, that makes sense to me. Sorry for the bikeshedding. > > Ryanne > > On Tue, Jul 16, 2019 at 12:49 PM John Roesler wrote: > > > Thanks for the explanation and the su

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

2019-07-16 Thread John Roesler
ntConfigs` class, > > I’ll go ahead and do that then. > > > > Thank you for your input! > > > > Best, > > Daniyar Yeralin > > > >> On Jul 15, 2019, at 11:45 AM, John Roesler wrote: > >> > >> Hi all, > >> > >&

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

2019-07-16 Thread John Roesler
r to "running", I'd suggest the more correct "moving", as in "moving > average" and "moving sum", which involve looking back N samples, applying a > sliding window, decaying over time etc. > > Ryanne > > On Tue, Jul 16, 2019, 11:58 AM Joh

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

2019-07-16 Thread John Roesler
en that "sum", "total", and "count" are roughly synonymous... > > In particular, TotalSum is, I think, a "running total", though the naming > doesn't really capture that. > > I think to avoid confusion, we should define exactly what "

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

2019-07-16 Thread John Roesler
PM John Roesler wrote: > > Hey all, > > It sounds like there's general agreement now on this KIP, so I updated > the KIP to fit in with Guozhang's overall proposed package structure. > Specifically, the proposed name for th

[VOTE] KIP-478 Strongly Typed Streams Processor API

2019-07-16 Thread John Roesler
Hi Dev, After a good discussion, I'd like to start the vote for KIP-478 (https://cwiki.apache.org/confluence/x/2SkLBw). The proposal is to deprecate the existing interface org.apache.kafka.streams.processor.Processor in favor of a new one, org.apache.kafka.streams.processor.api.Processor that par

Re: [DISCUSS] KIP-479: Add Materialized to Join

2019-07-16 Thread John Roesler
gt; > > implementation but with a more suitable interface than the current > > > WindowStore interface, how to do that is less clear and execution-wise > > it's > > > (arguably..) not urgent; 2) we want to close the last gap (Stream-stream > > > jo

Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams

2019-07-16 Thread John Roesler
put. > Regards > Patrik > > > Am 15.07.2019 um 18:03 schrieb John Roesler : > > > > Hey Patrik, > > > > Since KIP-471 is already accepted, and since this idea is not a > > trivial extension of the KIP, I think we'd need to do a new KIP.

Re: Behaviour of KStreamKTableJoinProcessor

2019-07-16 Thread John Roesler
again for my multiple replies. -John On Mon, Jul 15, 2019 at 11:35 AM John Roesler wrote: > > Hi again, Ties, > > I think I spoke too soon and also misread your email. > > By any chance, are you doing a join of a KStream and a GlobalKTable? > > In this case, it would ma

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

2019-07-15 Thread John Roesler
yped > >> > records, which I think would be acceptable. > >>3. Then we could tackle other pieces in an incremental manner as we see > >>what makes sense > >> > >> Just my 2cents > >> > >> -Bill > >> > >>

Re: Behaviour of KStreamKTableJoinProcessor

2019-07-15 Thread John Roesler
duce the memory/storage requirements, as each node won't have to host the whole global data set anymore. Please feel free to share your code in some form to clear up the situation in case I got it wrong again. Thanks, -John On Mon, Jul 15, 2019 at 10:48 AM John Roesler wrote: > > Hi Tie

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

2019-07-15 Thread John Roesler
MeasurableStat which itself is inherited by most of the classes you > mentioned in the KIP. Is it worth it converging on those as well (perhaps > deprecating/removing Measurable and opting for Gauge) or do we prefer to > keep the scope of this KIP small? > > Thanks, > Stanislav

Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams

2019-07-15 Thread John Roesler
ozhang, > > > > > > thank you for your comments. > > > > > > @Guozhang could you please also vote on the voting thread so that we > > > have all votes in one place. > > > > > > @John, the only situation I can think of where a n

Re: Behaviour of KStreamKTableJoinProcessor

2019-07-15 Thread John Roesler
Hi Ties, You're on the right track. You need to use `KTable.map` ahead of the join to select the new key. This will allow Streams to make sure the data is correctly partitioned to perform the join. Thanks, -John On Mon, Jul 15, 2019 at 10:07 AM Ven, Ties Jens van de wrote: > > I recently starte

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

2019-07-15 Thread John Roesler
s, I do agree. That totally makes sense. The only thing is that it goes > against what Matthias suggested earlier: > "I think that ... `ListSerde` should have an default constructor and it > should be possible to pass in the `Class listClass` information via a > configuration. Othe

[VOTE] KIP-488: Clean up Sum,Count,Total Metrics

2019-07-12 Thread John Roesler
Hi Kafka devs, Yesterday, I proposed KIP-488 as a minor cleanup of some of our metric implementations. KIP-488: https://cwiki.apache.org/confluence/x/kkAyBw The change seems pretty uncontroversial, so I'm just going to open the vote now. Feel free to veto or just request more discussion if you

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

2019-07-12 Thread John Roesler
> > > the javadocs, which btw could also be improved in those classes. > > > > > > Best, > > > Bruno > > > > > > > > > > > > On Thu, Jul 11, 2019 at 8:25 PM Matthias J. Sax > > > wrote: > > >> > > &g

[DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

2019-07-11 Thread John Roesler
Hi Kafka devs, I'd like to propose KIP-488 as a minor cleanup of some of our metric implementations. KIP-488: https://cwiki.apache.org/confluence/x/kkAyBw Over time, iterative updates to these metrics has resulted in a pretty confusing little collection of classes, and I've personally been invol

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

2019-07-10 Thread John Roesler
; >> (we need two consecutive round-trips to the RHS). > >> >> > >> >> S-2: Remember/store if a previous result exists on LHS: for this case, > >> >> (a) is handled straightforward, (b) is handled by telling RHS to send > >> >> tombstone if pre

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

2019-07-09 Thread John Roesler
ogic of conditional serialization based on the type of an inner serde. > > Thank you! > > Best, > Daniyar Yeralin > > On Jun 26, 2019, at 11:44 AM, John Roesler wrote: > > Thanks for the update, Daniyar! > > In addition to specifying the config interface, can you a

Re: Nag!! KAFKA-8629 - Some feedback wanted

2019-07-09 Thread John Roesler
Hey Andy, Thanks for looking into this. I've been curious about it for some time. I'm glad to hear that the gap to get there is so small. You mentioned potentially switching off the JMX stuff with a config option. I'm not sure hiding the JMX features behind a config flag would be good enough... C

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

2019-07-09 Thread John Roesler
Hi John, > > > > > > Yeah I think we should not do all the repackaging as part of this KIP as > > > well (we can just do the movement of the Processor / ProcessorSupplier), > > > but I think we need to discuss the end goal here since otherwise we may &

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

2019-06-28 Thread John Roesler
>> solution. What I don't understand atm: for which case would we need to > >> send unnecessary tombstone? I thought that the `instruction` field helps > >> to avoid any unnecessary tombstone? Seems I a missing case? > >> > >> Also for my own understanding:

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

2019-06-27 Thread John Roesler
criptionWrapper and here > we are. > > Thanks for the thoughts, and the info on the right-key. That was > enlightening, though I can't think of a use-case for it *at this point in > time*. :) > > Adam > > > > On Thu, Jun 27, 2019 at 12:29 PM John Roesler wrote: >

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

2019-06-27 Thread John Roesler
follow. If they want to add a FK-join, they will need to rework their > code in the KStreams app and make a new release, since the underlying > topology would be different and new internal topics would need to be > created. In other words, I don't think a rolling upgrade where the u

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

2019-06-26 Thread John Roesler
riptionResponseWrapper? It sort of looks like > > it's always going to be equal to (RHS-result != null). > > I believe you are correct, and I missed the forest for the trees. They are > > effectively the same thing, and I can simply remove the flag. I will code > > it up

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

2019-06-26 Thread John Roesler
ng (k,null) whenever we emit > an event that lacks a matching foreign key on the RHS, except in the > (k,null) -> (k,fk) case. > If this LHS oldValue=null, we know we would have emitted a deletion and so > (k,null) would be emitted out of the join. In this case we don't need

[DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

2019-06-26 Thread John Roesler
Hi Adam, Thanks for the proposed revision to your KIP (https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=74684836&selectedPageVersions=77&selectedPageVersions=74) in response to the concern pointed out during code review (https://github.com/apache/kafka/pull/5527#issuecom

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

2019-06-26 Thread John Roesler
his in the KIP > >>> discussion. > >>> > >>> > >>> Can you also update the KIP? > >>> > >>> > >>> > >>> -Matthias > >>> > >>> > >>> > >>> > >>>

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

2019-06-24 Thread John Roesler
- > > This is a very wild thought and I can totally understand if people feel > this is too much since it definitely enlarges the scope of this KIP a lot > :) just trying to play a devil's advocate here to do major refactoring and > avoid renaming Processor classes. > > &g

Re: [DISCUSS] KIP-479: Add Materialized to Join

2019-06-24 Thread John Roesler
ndowed bytes store, then we would need to change that if we want to > replace it with a different implementation interface. > > > Guozhang > > On Mon, Jun 24, 2019 at 8:59 AM John Roesler wrote: > > > Hey Guozhang and Bill, > > > > For what it's worth

Re: [DISCUSS] KIP-479: Add Materialized to Join

2019-06-24 Thread John Roesler
" at Materialized is the best approach. > > > Here I think I agree with John, that generally speaking it's better be a > > > control function on the `KTable` itself, rather than on `Materialized`, > > so > > > fixing it via adding functions th

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

2019-06-21 Thread John Roesler
cordProcessor. Thanks, -John On Fri, Jun 21, 2019 at 6:42 PM John Roesler wrote: > > Hi all, > > I've updated the KIP with the feedback so far. > > The naming question is still the biggest (only?) outstanding issue. It > would be good to hear some more thoughts on it.

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

2019-06-21 Thread John Roesler
one for changing the interface to TypedProcessor (as in the PoC), and one for just changing the Processor interface in-place, breaking source compatibility. How can we resolve this decision? Thanks, -John On Thu, Jun 20, 2019 at 5:44 PM John Roesler wrote: > > Thanks for the feedb

[jira] [Created] (KAFKA-8582) Consider adding an ExpiredWindowRecordHandler to Suppress

2019-06-21 Thread John Roesler (JIRA)
John Roesler created KAFKA-8582: --- Summary: Consider adding an ExpiredWindowRecordHandler to Suppress Key: KAFKA-8582 URL: https://issues.apache.org/jira/browse/KAFKA-8582 Project: Kafka Issue

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

2019-06-21 Thread John Roesler
; type size, but users could specify the type on the deserializer (via a > config again)? > > > About generics: nesting could be arbitrarily deep. Hence, I doubt we can > support this and a cast will be necessary at some point in the user code. > > > > -Matthias > > &g

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

2019-06-20 Thread John Roesler
pe then my next question would be, > > how much breakage we would introducing if we just modify the Processor > > signature directly? My feeling is that DSL users would be most likely not > > affected and PAPI users only need to modify a few lines on class > > declaration. I fee

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

2019-06-20 Thread John Roesler
my idea looks a little bit over engineered > :) > > I also wanted to see a feedback from Mathias as well since he gave me an > idea about storing fixed/variable size entries. > > Best, > Daniyar Yeralin > > On Jun 18, 2019, at 6:06 PM, John Roesler wrote: > >

Re: [DISCUSS] KIP-439: Deprecate Interface WindowStoreIterator

2019-06-20 Thread John Roesler
d also prefix the above names with `get-` if you think, it > >>> would make the names clearer. > >>> > >>> 3. Should `[storeType]-id` be `[storeType]-state-id`? > >>> > >>> 4. Some method signatures in the KIP under com

[jira] [Resolved] (KAFKA-8452) Possible Suppress buffer optimization: de-duplicate prior value

2019-06-20 Thread John Roesler (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-8452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler resolved KAFKA-8452. - Resolution: Fixed > Possible Suppress buffer optimization: de-duplicate prior va

Re: [VOTE] KIP-474: To deprecate WindowStore#put(key, value)

2019-06-20 Thread John Roesler
Not that it changes the outcome, but I'm also +1 (nonbinding). Been wanting to do this for a while, thanks Omkar! Now, if we can just get one more binding vote... -John On Sun, Jun 9, 2019 at 1:04 AM omkar mestry wrote: > > Hi, > > Ok the voting thread is still open. > > Thanks Regards > Omkar

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

2019-06-19 Thread John Roesler
Hi all, In response to the feedback so far, I changed the package name from `processor2` to `processor.generic`. Thanks, -John On Mon, Jun 17, 2019 at 4:49 PM John Roesler wrote: > > Thanks for the feedback, Sophie! > > I actually felt a little uneasy when I wrote that remark,

Re: [DISCUSS] KIP-479: Add Materialized to Join

2019-06-19 Thread John Roesler
Hi Bill, Thanks for the KIP! Awesome job catching this unexpected consequence of the prior KIPs before it was released. The proposal looks good to me. On top of just fixing the problem, it seems to address two other pain points: * that naming a state store automatically causes it to become queria

Re: [VOTE] KIP-471: Expose RocksDB Metrics in Kafka Streams

2019-06-19 Thread John Roesler
I'm +1 (nonbinding) Thanks! -John On Tue, Jun 18, 2019 at 7:48 AM Bruno Cadonna wrote: > > Hi, > > I would like to start the voting on KIP-471: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-471%3A+Expose+RocksDB+Metrics+in+Kafka+Streams > > You can find the discussion here: > https://l

Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams

2019-06-19 Thread John Roesler
now, -John On Wed, Jun 19, 2019 at 12:03 PM John Roesler wrote: > > Just taking a look over the metrics again, I had one thought... > > Stuff that happens in a background thread (like compaction metrics) > can't directly identify compactions as a bottleneck from Streams'

Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams

2019-06-19 Thread John Roesler
ics you proposed, but I'd suggest saying something to this effect in whatever documentation or descriptions we provide. Thanks, -John On Wed, Jun 19, 2019 at 11:25 AM John Roesler wrote: > > Thanks for the updates. > > Personally, I'd be in favor of not going out on a limb with

Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams

2019-06-19 Thread John Roesler
Thanks for the updates. Personally, I'd be in favor of not going out on a limb with unsupported metrics APIs. We should take care to make sure that what we add in KIP-471 is stable and well supported, even if it's not the complete picture. We can always do follow-on work to tackle complex metrics

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

2019-06-18 Thread John Roesler
Hi Daniyar, That's a very clever solution! One observation is that, now, this is what we might call a polymorphic serde. That is, you're detecting the actual concrete type and then promising to produce the exact same concrete type on read. There are some inherent problems with this approach, whic

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

2019-06-17 Thread John Roesler
of in the last month. Your state store will probably be holding some more > complicated PurchaseHistory object (keyed by user), but your output is just > a > > I'm also not crazy about "processor2" as the package name ... not sure what > a better one w

Re: Posted a new article about Kafka Streams

2019-06-17 Thread John Roesler
Hi all, Thanks for the heads-up! I've added the article to my reading list. And thanks for the reminder about your KIP, Paul. I just visited the discussion and voting threads. Also, another plug, you might both be interested in my freshly-proposed KIP-478 (https://cwiki.apache.org/confluence/x/2

Re: [VOTE] KIP-401: TransformerSupplier/ProcessorSupplier StateStore connecting

2019-06-17 Thread John Roesler
I'm +1 (nonbinding) on the current iteration of the proposal. On Mon, May 27, 2019 at 1:58 PM Paul Whalen wrote: > > I spoke too early a month ago, but I believe the proposal is finalized now > and ready for voting. > > KIP: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97553

Re: [DISCUSS] KIP-401 TransformerSupplier/ProcessorSupplier enhancements

2019-06-17 Thread John Roesler
Hey, all, Sorry I'm late to the party. I meant to read into this KIP before, but didn't get around to it. I was just reminded when Paul mentioned it in a different thread. Please feel free to bump a discussion any time it stalls! I've just read through the whole discussion so far, and, to echo th

Re: [DISCUSS] KIP-470: TopologyTestDriver test input and output usability improvements

2019-06-17 Thread John Roesler
o")), > hasProperty("headers", equalTo(headers; > > > Jukka > > la 15. kesäk. 2019 klo 1.10 John Roesler (j...@confluent.io) kirjoitti: > > > Sounds good. Thanks as always for considering my feedback! > > -John > > > > O

[DISCUSS] KIP-478 Strongly Typed Processor API

2019-06-17 Thread John Roesler
Hi all, I'd like to propose KIP-478 (https://cwiki.apache.org/confluence/x/2SkLBw). This proposal would add output type bounds to the Processor interface in Kafka Streams, which enables static checking of a number of useful properties: * A processor B that consumes the output of processor A is ac

Re: [DISCUSS] KIP-470: TopologyTestDriver test input and output usability improvements

2019-06-14 Thread John Roesler
es = Arrays.asList( > "Apache Kafka Streams Example", > "Using Kafka Streams Test Utils", > "Reading and Writing Kafka Topic" > ); > inputTopic.pipeValueList(inputValues); > > > Let's check after the next iteration is it still

Re: [DISCUSS] KIP-470: TopologyTestDriver test input and output usability improvements

2019-06-14 Thread John Roesler
nt to leave client package untouched, I would modify the methods > with ClientRecord now in InputTopic and OutputTopic to pass in and out this > TestRecord instead. > In that case there would be possibility to add methods to TestRecord to > help ignore some fields in assertions like: >

Re: [DISCUSS] KIP-470: TopologyTestDriver test input and output usability improvements

2019-06-13 Thread John Roesler
Hey, all, maybe we can jump-start this discussion. I think this approach would be very ergonomic for testing, and would help reduce boilerplate in tests. The think I like most about it is that it mirrors the mental model that people already have from Kafka Streams, in which you write to an "input

Re: [DISCUSSION] KIP-418: A method-chaining way to branch KStream

2019-06-04 Thread John Roesler
; > -> Map > > > > // return map of all names sub-stream into current scope > > KBranchedStream#noDefaultBranch() > > -> Map > > > > > > > > Hence, for each sub-stream, the user can pick to add a name and return > > the branch "resul

[jira] [Created] (KAFKA-8478) Poll for more records before forced processing

2019-06-04 Thread John Roesler (JIRA)
John Roesler created KAFKA-8478: --- Summary: Poll for more records before forced processing Key: KAFKA-8478 URL: https://issues.apache.org/jira/browse/KAFKA-8478 Project: Kafka Issue Type

[jira] [Created] (KAFKA-8455) Add NothingSerde to Serdes

2019-05-31 Thread John Roesler (JIRA)
John Roesler created KAFKA-8455: --- Summary: Add NothingSerde to Serdes Key: KAFKA-8455 URL: https://issues.apache.org/jira/browse/KAFKA-8455 Project: Kafka Issue Type: Improvement

[jira] [Created] (KAFKA-8452) Possible Suppress buffer optimization: de-duplicate prior value

2019-05-30 Thread John Roesler (JIRA)
John Roesler created KAFKA-8452: --- Summary: Possible Suppress buffer optimization: de-duplicate prior value Key: KAFKA-8452 URL: https://issues.apache.org/jira/browse/KAFKA-8452 Project: Kafka

[jira] [Resolved] (KAFKA-7991) Add StreamsUpgradeTest for 2.2 release

2019-05-28 Thread John Roesler (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-7991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler resolved KAFKA-7991. - Resolution: Duplicate Yep. Closing as a duplicate. > Add StreamsUpgradeTest for 2.2 rele

[jira] [Created] (KAFKA-8410) Strengthen the types of Processors, at least in the DSL, maybe in the PAPI as well

2019-05-22 Thread John Roesler (JIRA)
John Roesler created KAFKA-8410: --- Summary: Strengthen the types of Processors, at least in the DSL, maybe in the PAPI as well Key: KAFKA-8410 URL: https://issues.apache.org/jira/browse/KAFKA-8410

[jira] [Created] (KAFKA-8403) Suppress needs a Materialized variant

2019-05-21 Thread John Roesler (JIRA)
John Roesler created KAFKA-8403: --- Summary: Suppress needs a Materialized variant Key: KAFKA-8403 URL: https://issues.apache.org/jira/browse/KAFKA-8403 Project: Kafka Issue Type: Improvement

Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams

2019-05-20 Thread John Roesler
Hi Bruno, Looks really good overall. This is going to be an awesome addition. My only thought was that we have "bytes-flushed-(rate|total) and flush-time-(avg|min|max)" metrics, and the description states that these are specifically for Memtable flush operations. What do you think about calling i

[jira] [Created] (KAFKA-8396) Clean up Transformer API

2019-05-20 Thread John Roesler (JIRA)
John Roesler created KAFKA-8396: --- Summary: Clean up Transformer API Key: KAFKA-8396 URL: https://issues.apache.org/jira/browse/KAFKA-8396 Project: Kafka Issue Type: Improvement

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

2019-05-15 Thread John Roesler
Sounds good! On Tue, May 14, 2019 at 9:21 AM Development wrote: > > Hey, > > I think it the proposal is finalized, no one raised any concerns. Shall we > call it for a [VOTE]? > > Best, > Daniyar Yeralin > > > On May 10, 2019, at 10:17 AM, John Roesler wrote:

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

2019-05-10 Thread John Roesler
t 7:27 AM Development wrote: > >> > >>> Hey, > >>> > >>> I don’t see any replies. Seems like this proposal can be finalized and > >>> called for a vote? > >>> > >>> Also I’ve been thinking. Do we need more serdes for other Co

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

2019-05-08 Thread John Roesler
ted JIRA and KIP. > > > > I didn’t know about the process, and created PR before I knew about KIPs :) > > > > As per static declaration, I don’t think Java allows that: > > > > > > Best, > > Daniyar Yeralin > > > >> On May 7, 2019, at 2:22 PM, John Roe

Re: [DISCUSS] KIP-456: Helper classes to make it simpler to write test logic with TopologyTestDriver

2019-05-08 Thread John Roesler
Hi Jukka, thanks for the reply! I think this is a good summary (the discussion was getting a little unwieldy. I'll reply inline. Also, thanks for clarify about your library vs. this KIP. That makes perfect sense to me. > > 1. Add JavaDoc for KIP > > Is there a good example of KIP where Javadoc is

Re: [DISCUSS] KIP-456: Helper classes to make it simpler to write test logic with TopologyTestDriver

2019-05-07 Thread John Roesler
Thanks for the responses, Jukka! Thanks for the reference to the javadocs in your library, but I actually meant they should be part of the KIP document. As a general comment, I did get that your intent is to smooth over some rough spots with the current testing library, and that many of your API/

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

2019-05-07 Thread John Roesler
tatic > org.apache.kafka.common.serialization.Serdes.ListSerde) > > > On May 7, 2019, at 11:50 AM, John Roesler wrote: > > > > Thanks for the reply Daniyar, > > > > That makes much more sense! I thought I must be missing something, but I > > couldn't f

Re: [DISCUSS] 2.2.1 Bug Fix Release

2019-05-07 Thread John Roesler
On Fri, May 3, 2019 at 3:16 PM Vahid Hashemian > > wrote: > > > > > Hi John, > > > > > > Thanks for confirming. > > > I'll wait for final bug fix PR for this issue to get merged so we can > > > safely resolve the ticket. That makes

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

2019-05-07 Thread John Roesler
change log topic). > > Thank you! > > Best, > Daniyar Yeralin > > > > On May 6, 2019, at 5:37 PM, John Roesler wrote: > > > > Hi Daniyar, > > > > Thanks for the proposal! > > > > If I understand the point about the comparator, is it just t

<    4   5   6   7   8   9   10   11   12   >