[jira] [Resolved] (KAFKA-5242) add max_number _of_retries to exponential backoff strategy

[ https://issues.apache.org/jira/browse/KAFKA-5242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy resolved KAFKA-5242. --- Resolution: Won't Do This has been fixed by KAFKA-5152 which has already been merged to 0.11.0.1

Re: [DISCUSS] 0.11.0.1 bug fix release

. What's the ballpark when 0.11.0.1 will be available? > -Srikanth > > On Wed, Aug 16, 2017 at 5:59 PM, Damian Guy <damian@gmail.com> wrote: > > > Hi, > > > > It seems like it must be time for 0.11.0.1 bug fix release! > > > > Since the 0.11.0.0 releas

[DISCUSS] 0.11.0.1 bug fix release

Hi, It seems like it must be time for 0.11.0.1 bug fix release! Since the 0.11.0.0 release we've fixed 30 JIRAs that are targeted for 0.11.0.1:

[jira] [Resolved] (KAFKA-5440) Kafka Streams report state RUNNING even if all threads are dead

[ https://issues.apache.org/jira/browse/KAFKA-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy resolved KAFKA-5440. --- Resolution: Duplicate > Kafka Streams report state RUNNING even if all threads are d

[jira] [Resolved] (KAFKA-4643) Improve test coverage of StreamsKafkaClient

[ https://issues.apache.org/jira/browse/KAFKA-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy resolved KAFKA-4643. --- Resolution: Fixed Fix Version/s: 1.0.0 Issue resolved by pull request 3663 [https://github.com

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

lueSerde) public static <K, V> Produced<K, V> keySerde(final Serde keySerde) public static <K, V> Produced<K, V> valueSerde(final Serde valueSerde) public static <K, V> Produced<K, V> streamPartitioner(final StreamPartitioner<K, V> partitioner) So if you

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

constructors as `with` but I'm > not > > sure if Java distinguish: > > > > public static <K, V> Produced<K, V> with(final Serde keySerde) > > public static <K, V> Produced<K, V> with(final Serde valueSerde) > > > > as two function signatures. &

[jira] [Resolved] (KAFKA-5717) [streams] 'null' values in state stores

[ https://issues.apache.org/jira/browse/KAFKA-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy resolved KAFKA-5717. --- Resolution: Fixed Fix Version/s: 0.11.0.1 1.0.0 Issue resolved by pull

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

replaced by > BytesStoreSupplier? Seems some more descriptions are lacking here. Also in > > No modifications to StateStoreSupplier. It is superseceded by BytesStoreSupplier. > public static <K, V, S extends StateStore> Materialized<K, V, S> > as(final StateStoreSupplier > supplie

[jira] [Created] (KAFKA-5702) Refactor StreamThread to separate concerns and enable better testability

Damian Guy created KAFKA-5702: - Summary: Refactor StreamThread to separate concerns and enable better testability Key: KAFKA-5702 URL: https://issues.apache.org/jira/browse/KAFKA-5702 Project: Kafka

Re: KStreamPrintTest : no differences in the unit tests

Yes - they are basically the same. Feel free to submit a patch to remove one of them On Wed, 2 Aug 2017 at 15:28 Paolo Patierno wrote: > Hi devs, > > taking a look at KStreamPrintTest I can't find any substantial difference > between the two tests : > > >

[jira] [Created] (KAFKA-5689) Refactor WindowStore hierarchy so that Metered Store is the outermost store

Damian Guy created KAFKA-5689: - Summary: Refactor WindowStore hierarchy so that Metered Store is the outermost store Key: KAFKA-5689 URL: https://issues.apache.org/jira/browse/KAFKA-5689 Project: Kafka

[jira] [Resolved] (KAFKA-5154) Kafka Streams throws NPE during rebalance

[ https://issues.apache.org/jira/browse/KAFKA-5154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy resolved KAFKA-5154. --- Resolution: Fixed > Kafka Streams throws NPE during rebala

Re: Kafka Streams debugging with "no fluent" API choice

Advisor > > Twitter : @ppatierno<http://twitter.com/ppatierno> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno> > Blog : DevExperience<http://paolopatierno.wordpress.com/> > > > > From: Damian Guy <dam

Re: Kafka Streams debugging with "no fluent" API choice

t; Paolo Patierno > Senior Software Engineer (IoT) @ Red Hat > Microsoft MVP on Windows Embedded & IoT > Microsoft Azure Advisor > > Twitter : @ppatierno<http://twitter.com/ppatierno> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno> > Blog

Re: Kafka Streams debugging with "no fluent" API choice

I don't know specifically why this is removed, however if you want to get the same functionality you can use peek, i.e: stream.map(...).peek(...).filter(..) You can log the key values out in the peek call. On Tue, 1 Aug 2017 at 11:48 Paolo Patierno wrote: > Hi guys, > > >

[jira] [Created] (KAFKA-5680) Don't materliaze physical state stores in KTable filter/map etc operations

Damian Guy created KAFKA-5680: - Summary: Don't materliaze physical state stores in KTable filter/map etc operations Key: KAFKA-5680 URL: https://issues.apache.org/jira/browse/KAFKA-5680 Project: Kafka

[jira] [Created] (KAFKA-5673) Refactor KeyValueStore hierarchy so that MeteredKeyValueStore is the outermost store

Damian Guy created KAFKA-5673: - Summary: Refactor KeyValueStore hierarchy so that MeteredKeyValueStore is the outermost store Key: KAFKA-5673 URL: https://issues.apache.org/jira/browse/KAFKA-5673 Project

[jira] [Created] (KAFKA-5672) Move measureLatencyNs from StreamsMetricsImpl to StreamsMetrics

Damian Guy created KAFKA-5672: - Summary: Move measureLatencyNs from StreamsMetricsImpl to StreamsMetrics Key: KAFKA-5672 URL: https://issues.apache.org/jira/browse/KAFKA-5672 Project: Kafka

[jira] [Resolved] (KAFKA-5549) Explain that `client.id` is just used as a prefix within Streams

[ https://issues.apache.org/jira/browse/KAFKA-5549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy resolved KAFKA-5549. --- Resolution: Fixed Fix Version/s: 1.0.0 Issue resolved by pull request 3544 [https://github.com

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

Updated link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-182%3A+Reduce+Streams+DSL+overloads+and+allow+easier+use+of+custom+storage+engines Thanks, Damian On Thu, 27 Jul 2017 at 13:09 Damian Guy <damian@gmail.com> wrote: > Hi, > > I've put together a KIP to ma

[DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

Hi, I've put together a KIP to make some changes to the KafkaStreams DSL that will hopefully allow us to: 1) reduce the explosion of overloads 2) add new features without having to continue adding more overloads 3) provide simpler ways for people to use custom storage engines and wrap them with

[jira] [Created] (KAFKA-5655) Add new API methods to KGroupedTable

Damian Guy created KAFKA-5655: - Summary: Add new API methods to KGroupedTable Key: KAFKA-5655 URL: https://issues.apache.org/jira/browse/KAFKA-5655 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-5653) Add new API methods to KTable

Damian Guy created KAFKA-5653: - Summary: Add new API methods to KTable Key: KAFKA-5653 URL: https://issues.apache.org/jira/browse/KAFKA-5653 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-5654) Add new API methods to KGroupedStream

Damian Guy created KAFKA-5654: - Summary: Add new API methods to KGroupedStream Key: KAFKA-5654 URL: https://issues.apache.org/jira/browse/KAFKA-5654 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-5652) Add new api methods to KStream

Damian Guy created KAFKA-5652: - Summary: Add new api methods to KStream Key: KAFKA-5652 URL: https://issues.apache.org/jira/browse/KAFKA-5652 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-5651) KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

Damian Guy created KAFKA-5651: - Summary: KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines Key: KAFKA-5651 URL: https://issues.apache.org/jira/browse/KAFKA-5651 Project

[jira] [Created] (KAFKA-5650) Provide a simple way for custom storage engines to use streams wrapped stores (KIP-182)

Damian Guy created KAFKA-5650: - Summary: Provide a simple way for custom storage engines to use streams wrapped stores (KIP-182) Key: KAFKA-5650 URL: https://issues.apache.org/jira/browse/KAFKA-5650

Re: [VOTE] KIP-167 (Addendum): Add interface for the state store restoration process

+1 On Tue, 25 Jul 2017 at 18:17 Sriram Subramanian wrote: > +1 > > On Fri, Jul 21, 2017 at 12:08 PM, Guozhang Wang > wrote: > > > +1 > > > > On Thu, Jul 20, 2017 at 11:00 PM, Matthias J. Sax > > > wrote: > > > > > +1 > > > > > >

Re: [VOTE]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

Hi Jeyhun, Feel free to close the vote. It has been accepted. Thanks, Damian On Mon, 17 Jul 2017 at 06:18 Guozhang Wang <wangg...@gmail.com> wrote: > +1. Thanks! > > On Sat, Jul 8, 2017 at 1:35 AM, Damian Guy <damian@gmail.com> wrote: > > > +1 > > On Fr

[jira] [Resolved] (KAFKA-3741) Allow setting of default topic configs via StreamsConfig

[ https://issues.apache.org/jira/browse/KAFKA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy resolved KAFKA-3741. --- Resolution: Fixed Fix Version/s: 0.11.1.0 Issue resolved by pull request 3459 [https

Re: [VOTE] KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs

Sorry, i mentioned Gouzhang twice in the vote. Actual votes: 3 binding (Guozhang, Damian, Ismael) 2 non-binding (Eno, Matthias) Thanks, Damian On Fri, 21 Jul 2017 at 10:00 Damian Guy <damian@gmail.com> wrote: > Hi, > The Vote for this KIP is now closed. > KIP-173 has been ac

Re: [VOTE] KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs

it's just that Gmail's > algorithm decided to place it under the same thread. This is a recurring > problem and it affects discoverability of vote threads for Gmail users (I > assume there are many such users). > > Ismael > > On Thu, Jul 20, 2017 at 1:32 AM, Damian Guy <damian..

Re: [DISCUSS] 2017 October release planning and release version

+1 on 1.0! Are we also going to move to java 8? I also think we should drop the Unstable annotations completely. Cheers, Damian On Wed, 19 Jul 2017 at 21:36 Guozhang Wang wrote: > Hi Stevo, > > Just trying to add to what Ismael has already replied you: > > > >

Re: [VOTE] KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs

at the existing votes are still > counted. > > > > Ismael > > > > On Mon, Jul 17, 2017 at 4:43 AM, Damian Guy <damian@gmail.com> > wrote: > > > > > Hi, > > > > > > I'd like to kick off the vote for KIP-173: > > > h

Re: KIP-162: Enable topic deletion by default

+1 On Wed, 31 May 2017 at 13:36 Jim Jagielski wrote: > +1 > > On May 27, 2017, at 9:27 PM, Vahid S Hashemian < > vahidhashem...@us.ibm.com> wrote: > > > > Sure, that sounds good. > > > > I suggested that to keep command line behavior consistent. > > Plus, removal of ACL access

Re: [VOTE] KIP-162: Enable topic deletion by default

+1 On Tue, 6 Jun 2017 at 18:19 BigData dev wrote: > +1 (non-binding) > > Thanks, > Bharat > > On Tue, Jun 6, 2017 at 9:21 AM, Ashwin Sinha > wrote: > > > +1 > > > > On Tue, Jun 6, 2017 at 11:20 PM, Mickael Maison < > mickael.mai...@gmail.com> >

[VOTE] KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs

Hi, I'd like to kick off the vote for KIP-173: https://cwiki.apache.org/confluence/display/KAFKA/KIP-173%3A+Add+prefix+to+StreamsConfig+to+enable+setting+default+internal+topic+configs A PR for this can be found here: https://github.com/apache/kafka/pull/3459 Thanks, Damian

Re: [DISCUSS] KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs

As switching to AdminClient does not require a KIP, we can save one > KIP. > > But I am fine either way if you want to exclude it. > > -Matthias > > On 7/15/17 1:59 PM, Damian Guy wrote: > > Thanks Matthias. I'll have a look into it. Though i guess we are planning > &

Re: [DISCUSS] KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs

admin." prefix, too. For configs > for StreamsKafkaClient and later AdminClient. > > Can so picky back this on this KIP or should we do a new KIP? > > > -Matthias > > On 7/10/17 6:45 PM, Guozhang Wang wrote: > > Thanks Damian. LGTM. > > > > Guozhan

Re: [VOTE] KIP-167: Add interface for the state store restoration process

+1 On Thu, 13 Jul 2017 at 07:13 Eno Thereska wrote: > +1 (non-binding). > > Thanks Bill. > > Eno > > On 12 Jul 2017, at 09:12, Bill Bejeck wrote: > > > > All, > > > > Now that we've concluded a second round of discussion on KIP-167, I'd > like > > to

Re: [ANNOUNCE] New Kafka PMC member Jason Gustafson

Congratulations Jason On Wed, 12 Jul 2017 at 08:57 Sriram Subramanian wrote: > Congratulations Jason! > > On Wed, Jul 12, 2017 at 8:02 AM, Rajini Sivaram > wrote: > > > Congratulations, Jason! > > > > On Wed, Jul 12, 2017 at 3:53 PM, Ismael Juma

Re: [DISCUSS] KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs

than manually change the web docs for > educating users. > > > Guozhang > > > On Fri, Jun 30, 2017 at 4:18 AM, Damian Guy <damian@gmail.com> wrote: > > > Hi, > > > > I've put together what will hopefully be a not too contentious KIP to > > enable the se

[jira] [Resolved] (KAFKA-5157) Options for handling corrupt data during deserialization

[ https://issues.apache.org/jira/browse/KAFKA-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy resolved KAFKA-5157. --- Resolution: Fixed Issue resolved by pull request 3423 [https://github.com/apache/kafka/pull/3423

[jira] [Created] (KAFKA-5578) Streams Task Assignor should consider the staleness of state directories when allocating tasks

Damian Guy created KAFKA-5578: - Summary: Streams Task Assignor should consider the staleness of state directories when allocating tasks Key: KAFKA-5578 URL: https://issues.apache.org/jira/browse/KAFKA-5578

Re: [VOTE]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

+1 On Fri, 7 Jul 2017 at 16:08, Eno Thereska wrote: > +1 (non-binding) Thanks. > > Eno > > On 6 Jul 2017, at 21:49, Gwen Shapira wrote: > > > > +1 > > > > On Wed, Jul 5, 2017 at 9:25 AM Matthias J. Sax > > wrote: > > > >> +1 >

Re: Possible deadlock during shutdown in setState in kafka streams 10.2

Thanks On Fri, 7 Jul 2017 at 18:20, Greg Fodor <gfo...@gmail.com> wrote: > Sure thing: https://issues.apache.org/jira/browse/KAFKA-5571 > > On Fri, Jul 7, 2017 at 2:59 AM, Damian Guy <damian@gmail.com> wrote: > > > Hi Greg, > > > > Would you mind cre

[jira] [Resolved] (KAFKA-5566) Instable test QueryableStateIntegrationTest.shouldAllowToQueryAfterThreadDied

[ https://issues.apache.org/jira/browse/KAFKA-5566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy resolved KAFKA-5566. --- Resolution: Fixed Fix Version/s: 0.11.0.1 0.11.1.0 Issue resolved by pull

Re: Possible deadlock during shutdown in setState in kafka streams 10.2

Hi Greg, Would you mind creating a JIRA for this with the thread dump ( i don't see it attached to your message). Thanks, Damian On Fri, 7 Jul 2017 at 10:36 Greg Fodor wrote: > I'm running a 10.2 job across 5 nodes with 32 stream threads on each node > and find that when

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

replacing the overrides. Thanks, Damian > About ProcessorContext vs RecordContext, you are right. I think I need to > implement a prototype to understand the full picture as some parts of the > KIP might not be as straightforward as I thought. > > > Cheers, > Jeyhun > > On Wed

Re: RocksDB flushing issue on 0.10.2 streams

Jul 2017 at 12:43 Damian Guy <damian@gmail.com> wrote: > Greg, what OS are you running on? > Are you able to reproduce this in a test at all? > For instance, based on what you described it would seem that i should be > able to start a streams app, wait for it to be u

Re: RocksDB flushing issue on 0.10.2 streams

, 5 Jul 2017 at 23:23 Damian Guy <damian@gmail.com> wrote: > Thanks Greg. I'll look into it more tomorrow. Just finding it difficult to > reproduce in a test. > Thanks for providing the sequence, gives me something to try and repo. > Appreciated. > > Thanks, > Damian

[jira] [Created] (KAFKA-5562) Do streams state directory cleanup on a single thread

Damian Guy created KAFKA-5562: - Summary: Do streams state directory cleanup on a single thread Key: KAFKA-5562 URL: https://issues.apache.org/jira/browse/KAFKA-5562 Project: Kafka Issue Type

[jira] [Resolved] (KAFKA-5508) Documentation for altering topics

[ https://issues.apache.org/jira/browse/KAFKA-5508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy resolved KAFKA-5508. --- Resolution: Fixed Fix Version/s: 0.11.0.1 0.11.1.0 Issue resolved by pull

Re: [ANNOUNCE] New Kafka PMC member Ismael Juma

Congratulations Ismael! Very well deserved. Cheers, Damian On Wed, 5 Jul 2017 at 22:54, Dong Lin wrote: > Congratulations Ismael! > > On Wed, Jul 5, 2017 at 1:55 PM, Jun Rao wrote: > > > Hi, Everyone, > > > > Ismael Juma has been active in the Kafka

Re: RocksDB flushing issue on 0.10.2 streams

ently running + assigned tasks (perhaps during > a > > rebalance there is a race condition?) I'm guessing there's probably a > more > > benign explanation, but that is what it looks like right now. > > > > On Wed, Jul 5, 2017 at 7:00 AM, Damian Guy <damian@gmail.com&

Re: RocksDB flushing issue on 0.10.2 streams

BTW - i'm trying to reproduce it, but not having much luck so far... On Wed, 5 Jul 2017 at 09:27 Damian Guy <damian@gmail.com> wrote: > Thans for the updates Greg. There were some minor changes around this in > 0.11.0 to make it less likely to happen, but we've only ever seen t

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

HI Jeyhun, Is the intention that these methods are new overloads on the KStream, KTable, etc? It is worth noting that a ProcessorContext is not a RecordContext. A RecordContext, as it stands, only exists during the processing of a single record. Whereas the ProcessorContext exists for the

Re: RocksDB flushing issue on 0.10.2 streams

t 5-10 minutes across 5 nodes before this happened. Does the log > show a > >>> rebalance was happening? It looks to me like the standby task was just > >>> committing as part of normal operations. > >>> > >>> On Tue, Jul 4, 2017 at 7:40 AM, Dami

[jira] [Created] (KAFKA-5556) KafkaConsumer throws: java.lang.IllegalStateException: > Attempt to retrieve exception from future which hasn't failed

Damian Guy created KAFKA-5556: - Summary: KafkaConsumer throws: java.lang.IllegalStateException: > Attempt to retrieve exception from future which hasn't failed Key: KAFKA-5556 URL: https://issues.apache.org/j

Re: RocksDB flushing issue on 0.10.2 streams

Hi Greg, Obviously a bit difficult to read the RocksDBException, but my guess is it is because the state directory gets deleted right before the flush happens: 2017-07-04 10:54:46,829 [myid:] - INFO [StreamThread-21:StateDirectory@213] - Deleting obsolete state directory 0_10 for task 0_10 Yes

Re: [DISCUSS] Streams DSL/StateStore Refactoring

are talking about. I would argue to go a bit slower and more carefull on > this one. At some point we need to get it right. Peeking over to the hadoop > guys with their hughe userbase. Config files really work well for them. > > Best Jan > > > > > > On 30.06.2017 09:31

Re: [DISCUSS] Streams DSL/StateStore Refactoring

t -- not necessarily a operator/window question. > > Sure, within a sub-topolgy it makes sense for the window retention to be the same. However, different sub-toplogies may have different constraints. > > -Matthias > > On 6/30/17 12:31 AM, Damian Guy wrote: > > Thanks Matthia

Re: [DISCUSS] KIP-167: Add interface for the state store restoration process

Thanks for the updated KIP Bill. In the PR you have AbstractBatchingRestoreCallback and AbstractNotifyingRestoreCallback which are both in public packages, so are part of the API. I think you need to add those to the KIP to round it off. Otherwise LGTM. Thanks, Damian On Fri, 30 Jun 2017 at

[DISCUSS] KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs

Hi, I've put together what will hopefully be a not too contentious KIP to enable the setting of default configs for streams internal topics via StreamsConfig. You can find the KIP here:

Re: [DISCUSS] Streams DSL/StateStore Refactoring

tream.groupBy().count() As i said above, everything that happens before the final aggregate call can be applied to any of them. So it makes sense to me to do those things ahead of the final aggregate call. > Last about builder pattern. I am convinced that we need some "terminal" > opera

Re: [VOTE] KIP 130: Expose states of active tasks to KafkaStreams public API

been so long on this KIP. > >> > >> Thanks. > >> > >> 2017-04-24 18:56 GMT+02:00 Guozhang Wang <wangg...@gmail.com>: > >> > >>> Florian, could you also add the part of deprecating > >> `KafkaStreams.toString` > >>> in

Re: [DISCUSS] KIP-171: Extend Consumer Group Reset Offset for Stream Application

Hi, Thanks for the KIP. What is not clear is how is this going to handle state stores? Right now the streams reset tool, resets everything and clears up the state stores. What are we going to do if we reset to a particular offset? If we clear the state then we've lost any previously aggregated

Re: [DISCUSS] Streams DSL/StateStore Refactoring

is reusable and if they attempted to reuse the Count > for some reason it would throw an error message saying that a store named > “my-store” already exists. > > Yes i agree and i think using builders is my preferred pattern. Cheers, Damian > Thanks, > Kyle > > From: Damian

Re: [DISCUSS] Streams DSL/StateStore Refactoring

one.join(Joins.streamStreamJoin(two, vj, JoinWindows.of(10)).build()); //left one.join(Joins.streamStreamJoin(two, vj, JoinWindows.of(10)).withJoinType(JoinType.LEFT).build()); } I'm not going to say which way i'm leaning, yet! Thanks, Damian On Thu, 29 Jun 2017 at 11:47 Dam

Re: [DISCUSS] Streams DSL/StateStore Refactoring

tream. Then the final count()/reduce()/aggregate() call has any of the params that are required/specific to that function. > > -Matthias > > On 6/28/17 2:55 AM, Damian Guy wrote: > >> I also think that mixing optional parameters with configs is a bad idea. > >> Have n

Re: [DISCUSS] Streams DSL/StateStore Refactoring

LoggingTopicConfig()"? > > > > Guozhang > > > > On Wed, Jun 28, 2017 at 3:59 PM, Matthias J. Sax <matth...@confluent.io> > wrote: > > > I see your point about "when to add the processor to the topology". That > > is indeed an issue. Not sure it

Re: [DISCUSS] Streams DSL/StateStore Refactoring

em with having all the withXX() in KTable interface > > -- but this might be subjective. > > > > > > However, I don't understand your argument about putting aggregate() > > after the withXX() -- all the calls to withXX() set optional parameters > > for aggregate()

Re: [DISCUSS] Streams DSL/StateStore Refactoring

uot;name").mapValues(...) > > Each .withXXX(...) return the current KTable and all the .withXXX() are > just added to the KTable interface. Or do I miss anything why this wont' > work or any obvious disadvantage? > > > See above. > > -Matthias > > On 6/22/17 4:0

[jira] [Resolved] (KAFKA-5487) Rolling upgrade test for streams

[ https://issues.apache.org/jira/browse/KAFKA-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy resolved KAFKA-5487. --- Resolution: Fixed Fix Version/s: (was: 0.11.0.1) 0.11.1.0 Issue

Re: Mirroring documentation improvement

From: Tom Bentley <t.j.bent...@gmail.com> > Sent: Friday, June 23, 2017 2:10 PM > To: dev@kafka.apache.org > Subject: Re: Mirroring documentation improvement > > Hi Damian, my username is tombentley > > Thanks > > Tom > > On 23 June 2017 at 14:57, Damian Guy <

Re: Mirroring documentation improvement

Evgeniy, you should now have access. Tom & Paolo what are your wiki usernames? Thanks, Damian On Fri, 23 Jun 2017 at 14:52 Tom Bentley wrote: > If any of the committers see this: Could I also have edit access please? > > On 23 June 2017 at 14:47, Evgeniy Veretennikov

Re: [VOTE] KIP-161: streams deserialization exception handlers

Thanks for the KIP Eno. +1 (binding) On Fri, 23 Jun 2017 at 11:00 Eno Thereska wrote: > Starting voting thread for: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-161%3A+streams+deserialization+exception+handlers > < >

[jira] [Resolved] (KAFKA-4913) creating a window store with one segment throws division by zero error

[ https://issues.apache.org/jira/browse/KAFKA-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy resolved KAFKA-4913. --- Resolution: Fixed Fix Version/s: (was: 0.11.0.1) 0.11.1.0 Issue

Re: Kafka streams KStream and ktable join issue

My guess is it is because the record doesn't have a key, i.e., the key is null. We have a fix for this in 0.11, in that we will skip records with a null key during restore. On Fri, 23 Jun 2017 at 03:57 Matthias J. Sax wrote: > Hi, > > can you reproduce the error reliably?

Re: [DISCUSS] Streams DSL/StateStore Refactoring

this case) that gives you what you actually care about (`KTable` in this > case). Not major issues, but worth mentioning while doing the comparison. > > The builder approach avoids the issues mentioned above, but it doesn't read > as well. > > Ismael > > On Wed, Jun 21,

Re: [DISCUSS] Streams DSL/StateStore Refactoring

duce the > number of overloads by quite a bit. > > Eno > > > On Jun 21, 2017, at 3:37 PM, Damian Guy <damian@gmail.com> wrote: > > > > Hi, > > > > I'd like to get a discussion going around some of the API choices we've > > made in the DLS. In

Re: confluence permission request

Hi, That should be done. Thanks, Damian On Wed, 21 Jun 2017 at 05:42 Kenji Hayashida wrote: > To Kafka Dev Team, > > Sorry, forgot sending my ID. > My ID is kenjih. > > Thanks. > > - Kenji Hayashida > > 2017-06-21 13:29 GMT+09:00 Kenji Hayashida : >

Re: Contributor

Done - thanks On Wed, 21 Jun 2017 at 12:19 Tom Bentley <t.j.bent...@gmail.com> wrote: > Please can I also be added? My username is tombentley. > > Thanks > > Tom > > On 21 June 2017 at 12:03, Damian Guy <damian@gmail.com> wrote: > > > Hi

[DISCUSS] Streams DSL/StateStore Refactoring

Hi, I'd like to get a discussion going around some of the API choices we've made in the DLS. In particular those that relate to stateful operations (though this could expand). As it stands we lean heavily on overloaded methods in the API, i.e, there are 9 overloads for KGroupedStream.count(..)!

Re: Contributor

Hi Andras, You should have access now. Thanks, Damian On Wed, 21 Jun 2017 at 10:45 Andras Beni wrote: > Hi All, > > I'd like to contribute to Apache Kafka. > Can you please add me (username: andrasbeni) to the contributors list for > this project at issues.apache.org?

Re: [ANNOUNCE] New committer: Damian Guy

; Well deserved. Congratulations Damian! > > > > On Fri, Jun 9, 2017 at 1:34 PM Guozhang Wang <wangg...@gmail.com> wrote: > > > > > Hello all, > > > > > > > > > The PMC of Apache Kafka is pleased to announce that we have invited > > Damian >

[jira] [Updated] (KAFKA-3741) KStream config for changelog min.in.sync.replicas

[ https://issues.apache.org/jira/browse/KAFKA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy updated KAFKA-3741: -- Status: Patch Available (was: Open) > KStream config for changelog min.in.sync.repli

[jira] [Assigned] (KAFKA-3741) KStream config for changelog min.in.sync.replicas

[ https://issues.apache.org/jira/browse/KAFKA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy reassigned KAFKA-3741: - Assignee: Damian Guy > KStream config for changelog min.in.sync.repli

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

/server.properties to 0 to improve > > the quickstart experience? > > > > Thanks, > > > > Jun > > > > > > On Tue, Apr 11, 2017 at 12:19 AM, Damian Guy <damian@gmail.com> > wrote: > > > > > Hi Onur, > > > > &g

Re: [DISCUSS] KIP-167: Add a restoreAll method to StateRestoreCallback

I'm largely in agreement with what Guozhang has suggested, i.e., StateRestoreContext shouldn't have any setters on it and also need to have the end offset available such that people can use it derive progress. Slightly different, maybe the StateRestoreContext interface could be: long

[jira] [Updated] (KAFKA-4913) creating a window store with one segment throws division by zero error

[ https://issues.apache.org/jira/browse/KAFKA-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy updated KAFKA-4913: -- Status: In Progress (was: Patch Available) > creating a window store with one segment throws divis

[jira] [Work stopped] (KAFKA-4913) creating a window store with one segment throws division by zero error

[ https://issues.apache.org/jira/browse/KAFKA-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-4913 stopped by Damian Guy. - > creating a window store with one segment throws division by zero er

Re: Sink Processor definition

Hi Michal, In this case Sink Processor is really referring to a SinkNode that can only produce to a kafka topic. Maybe the terminology is incorrect as strictly speaking a processor that writes data to anything could be considered a Sink Processor. On Sat, 3 Jun 2017 at 09:23 Michal Borowiecki

Re: [DISCUSS] KIP-147: Add missing type parameters to StateStoreSupplier factories and KGroupedStream/Table methods

Hmm, i guess this won't work due to adding the additional <K,V> to the StateStoreSupplier params on reduce, count, aggregate etc. On Sat, 3 Jun 2017 at 09:06 Damian Guy <damian@gmail.com> wrote: > Hi Michal, > > Thanks for the KIP - is there a way we can do this without

Re: [DISCUSS] KIP-147: Add missing type parameters to StateStoreSupplier factories and KGroupedStream/Table methods

Hi Michal, Thanks for the KIP - is there a way we can do this without having to introduce the new Typed.. Interfaces, overloaded methods etc? Is it possible that we just need to provide a couple of new methods on PersistentKeyValueFactory for windowed and sessionWindowed to return interfaces like

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

s would be the worst possible addition to the kafka > toolkit that I can think of. It just doesn't fit the architecture > of having clients falling behind is a valid option. > > Further. I mentioned already the only bad pill ive seen so far is crc > errors. any plans for those? > > Be

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

I agree with what Matthias has said w.r.t failing fast. There are plenty of times when you don't want to fail-fast and must attempt to make progress. The dead-letter queue is exactly for these circumstances. Of course if every record is failing, then you probably do want to give up. On Fri, 2

[jira] [Work started] (KAFKA-5357) StackOverFlow error in transaction coordinator

[ https://issues.apache.org/jira/browse/KAFKA-5357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-5357 started by Damian Guy. - > StackOverFlow error in transaction coordina

[jira] [Assigned] (KAFKA-5357) StackOverFlow error in transaction coordinator

[ https://issues.apache.org/jira/browse/KAFKA-5357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy reassigned KAFKA-5357: - Assignee: Damian Guy > StackOverFlow error in transaction coordina

<    1   2   3   4   5   6   7   8   >