-ZooKeeper-based consumers).
Warning: Consumer group 'GeoTest' is rebalancing.
I keep seeing the above lock exception continuously and app is not making
any progress. Any idea why it is stuck?
I read a few suggestions that required me to manually delete state
directory. I'd like to avoid that.
Thanks,
Srikanth
or few packages.
I assume any logging that indicates why app isn't making progress would be
> in WARN. I'll bump the level up for few days.
> Hope this helps.
>
>
> -Matthias
>
>
> On 12/5/17 9:35 AM, Srikanth wrote:
> > Hello,
> >
> > We
!
18/01/24 06:50:09 WARN StreamThread: Exception caught during
Deserialization, taskId: 0_0, topic: docker.event.1, partition: 0, offset:
3764
These incremented skipped-records-[rate|total].
Thanks,
Srikanth
Guozhang,
I read the comments in KAFKA-6376 and realized that we do something similar
to handle exception in process().
We register a custom metric "process-skipped" and our alerting system looks
for both this and skipped-records.
Srikanth
On Fri, Jan 26, 2018 at 10:26 PM, Guozhang W
ed by skippedDueToDeserializationError is missing.
Srikanth
On Fri, Jan 26, 2018 at 10:45 PM, Guozhang Wang wrote:
> Hi Srikanth,
>
> Looked at the source code once again and discussing with other committer I
> now remembered why we designed it that way: when you set the
> HandlerResponse to
Kafka-streams version "1.0.0".
Thanks,
Srikanth
On Mon, Jan 29, 2018 at 12:23 AM, Guozhang Wang wrote:
> Hello Srikanth,
>
> Which version of Kafka are you using? I'd like to dig for that particular
> branch again.
>
> Guozhang
>
> On Sun, J
Matthias,
We moved to streams client 1.0.0 with 0.10.2.x broker now. Haven't see this
issue ever since.
Thanks!
Srikanth
On Thu, Dec 7, 2017 at 12:15 AM, Matthias J. Sax
wrote:
> Running Streams 1.0.0 should just work with 0.10.2 brokers. Of course,
> you can't use EOS featu
ASS_CONFIG,
LogAndContinueExceptionHandler.class);
...
return p;
}
StreamsConfig streamsConfig = new StreamsConfig(getProperties())
KafkaStreams kafkaStreams = new KafkaStreams(streamBuilder.
build(),streamsConfig);
Srikanth
On Mon, Jan 29, 2018 at 11:10 PM, Guozhang Wang wrote:
> Hi Srikanth
Hi Guozhang,
You are right. I overlooked the fact that skippedDueToDeserializationError
is recorded as DEBUG.
That was it. Now that I got it, it feels like an overkill to set metrics
level to DEBUG just for this!
Thanks for your time!
Srikanth
On Tue, Jan 30, 2018 at 10:56 PM, Guozhang Wang
ome statistics on key distribution offline and use it in
the partitioner.
Is that a good idea? Or is it way too much logic for a partitioner?
Anything else to consider?
Any thoughts or reference will be helpful.
Thanks,
Srikanth
to do this is similar to Wesley. And consumers of topic can be
efficient if they get records from one(or a very few) keys.
Why do you think it is not applicable to Kafka? Are you suggesting that
there are other ways to handle it when using Kafka?
Srikanth
On Tue, May 3, 2016 at 11:58 AM, Tauzell,
te several uploads(one file per
key), upload file size vary, etc.
These can be solved in a few ways. One of them is to have kafka producer
partition by key. If I decide to do that then I have to solve the question
I posted first!!!
Srikanth
On Tue, May 3, 2016 at 1:22 PM, Tauzell, Dave
wrote:
r way. Other alternates I
thought off involves another step in processing.
Srikanth
On Tue, May 3, 2016 at 4:02 PM, Tauzell, Dave
wrote:
> Ok, I see what you are doing. Unless you have 1500 partitions and 1500
> consumers you will have consumers get records for different keys and will
> hav
Yeah, fixed slicing may help. I'll put more thought into this.
You had mentioned that you didn't put custom partitioner into production.
Would you mind sharing how you worked around this currently?
Srikanth
On Tue, May 3, 2016 at 5:43 PM, Wesley Chow wrote:
> >
> > Uploa
ution in our largest
> volume topic, and then squash the data and repartition based on a skewed
> key. The resulting skew is somewhat insignificant compared to our largest
> volume topic that we tend to not care.
>
> Wes
>
>
> > On May 4, 2016, at 2:57 PM, Srikanth wrot
.stream("PageViews")
.map(...)
.filter(...)
KTable users = builder.table("UserProfile")
.mapValues(...)
KStream regionCount = views
.leftJoin(user, ...)
.map(...)
.process(...)
I hope I was able to explain my questions clear enough for you to
understand.
Thanks,
Srikanth
f partitions.
Thanks,
Srikanth
On Thu, May 19, 2016 at 2:29 PM, Guozhang Wang wrote:
> Hello Srikanth,
>
> Thanks for your questions, please see replies inlined.
>
>
> On Tue, May 17, 2016 at 7:36 PM, Srikanth wrote:
>
> > Hi,
> >
> > I was reading ab
Thanks for the details!
I do see a pattern where through() is useful both explicitly and implicitly
by the DSL. I guess that fits well with kafka streams design of utilizing
kafka's strength.
Srikanth
On Fri, May 20, 2016 at 4:38 AM, Matthias J. Sax
wrote:
> Hi Srikanth,
>
> I b
m config at global level.
Timestamp extraction logic on the other hand will be specific to each
stream.
How does one write a generic extractor?
Thanks,
Srikanth
Matthias,
For (2), how do you achieve this using transform()?
Thanks,
Srikanth
On Sat, May 21, 2016 at 9:10 AM, Matthias J. Sax
wrote:
> Hi Srikanth,
>
> 1) there is no support on DSL level, but if you use Processor API you
> can do "anything" you like. So yes, a map
as reaching end of current content.
Matthias,
I haven't spent enough time on the approach you outlined. Will let you know.
Srikanth
On Mon, May 23, 2016 at 1:40 PM, Matthias J. Sax
wrote:
> Hi Srikanth,
>
> as Guozhang mentioned, the problem is the definition of the time, when
eed to never run in parallel?
The jira you gave seems to be on the same lines.
Can you comment on my question regarding TimestampExtractor?
We set one TimestampExtractor as a stream config at global level. Timestamp
extraction logic on the other hand will be specific to each stream.
Srikanth
On
gt; new
KeyValue(v._1, v._2))
override def extract(record: ConsumerRecord[Object, Object]): Long = {
record.topic match {
case rawLogTopicName => System.currentTimeMillis
case metadataTopicName => System.currentTimeMillis - 8640
case _ => System.currentTimeMillis
'd like to look at details for source & intermediate topics too.
Srikanth
.map((k,v) => new
KeyValue(v._1, v._2))
.to("intermediate topic")
Another topology will read the stream topic and perform a join.
val kTable = kStreamBuilder.table(intSerde, intSerde, "intermediate
topic")
val joinedKStream = someKStream.join(kTable, ...)
Any thoughts on what could be a good approach?
Srikanth
In terms of performance there is not going to be much difference to+table
vs through+aggregateByKey rt?
Srikanth
On Thu, Jun 2, 2016 at 9:21 AM, Matthias J. Sax
wrote:
> Hi Srikanth,
>
> your third approach seems to be the best fit. It uses only one shuffle
> of the data (whic
g out KafkaStreams for one new streaming app we are working on.
We'll most likely stick to DSL for that.
Does the DSL expose any stat or debug info? Or any way to access the
underlying Context?
Srikanth
On Thu, Jun 2, 2016 at 9:30 AM, Matthias J. Sax
wrote:
> Hi Srikanth,
>
> I am no
(100394<-null)
626026 , (100394<-null)
Finished Streams Example.
BTW, what is the strange "*<-null"* in KTable.print mean?
Srikanth
On Thu, Jun 2, 2016 at 9:21 AM, Matthias J. Sax
wrote:
> Hi Srikanth,
>
> your third approach seems to be the best fit. It uses only o
ollowing rt?
517631 , (100226<-null)
517631 , (100226<-100226)
Full code can be seen in my previous email.
Srikanth
On Fri, Jun 3, 2016 at 2:02 AM, Guozhang Wang wrote:
> Hello Srikanth,
>
> When involved in joins, KTable need to pass both the old value as well as
> the new value a
Thanks Matthias!
1) I didn't realize kafka-consumer-groups.sh only queries consumer
coordinator.
I was checking after terminating the streaming app. Got this via
console-consumer.
2) Understood.
3) Nope. Will check this out.
4)Yes, I can probably have a processorSupplier for this.
Sri
lear this store when time passes?
Srikanth
On Thu, Jun 30, 2016 at 4:27 AM, Clive Cox
wrote:
> Hi Eno,
> I've looked at KIP-67. It looks good but its not clear what calls I would
> make to do what I presently need: Get access to each windowed store at some
> time soon after win
some point we need to "complete" a window rt? Either based on processing
time or event time + watermark, etc.
How can we tie internal state store query with window completion? i.e, get
the final value.
Srikanth
On Thu, Jul 7, 2016 at 2:05 PM, Eno Thereska wrote:
> Hi Srikanth, Clive,
as an advantage when Table1 is much larger than Table2.
Each instance of the processor will have to hold entire Table2.
Option2 is to re-partition Table1 using through(StreamPartitioner) and
partition using value.
This will ensure co-location. Then join with Table2. This part might be
tricky??
Your comments and suggestions are welcome!
Srikanth
Matthias,
With option 2, how would we perform join after re-partition. Although we
re-partitioned with value, the key doesn't change.
KTable joins always use keys and ValueJoiner get values from both table
when keys match.
Having data co-located will not be sufficient rt??
Srikanth
On Thu
Michael,
Thanks! Looking forward to the update.
An interface like KTable is very conducive for joins. Hopefully, it will
get more flexible.
Srikanth
On Thu, Jul 14, 2016 at 4:35 AM, Michael Noll wrote:
> Srikant,
>
> > Its not a case for join() as the keys don't match.
be useful.
That's a lot of load but we still may miss the final computed value, which
is needed in most cases.
Srikanth
On Thu, Jul 14, 2016 at 5:04 AM, Michael Noll wrote:
> Srikanth,
>
> > This would be useful in place where we use a key-value store just to
> > duplicate a K
will
then override 111.
Srikanth
On Thu, Jul 14, 2016 at 11:07 AM, Matthias J. Sax
wrote:
> You will need to set a new key before you do the re-partitioning. In
> your case, it seems you want to switch key and value. This can be done
> with a simple map
>
> > table1.toStrea
d we don't have to replicate.
Srikanth
On Thu, Jul 14, 2016 at 11:36 AM, Matthias J. Sax
wrote:
> My bad... I should have considered this in the first place. You are
> absolutely right.
>
> Supporting this kind of a join is work in progress.
> https://issues.apache.org/jira/brow
I was looking for KTable-KTable semantics where both trigger updates.
The result will be used to enrich a few KStreams.
I'll keep an eye on this jira.
Meanwhile, I'll use custom processor or like you said convert it to
KStream-KTable join and continue with my test.
Srikanth
On T
posed to a callback that triggers
and at the right moment.
May be a Flink style trigger interface will help too.
Eager to see how it works in practice when released.
Srikanth
On Thu, Jul 14, 2016 at 12:36 PM, Guozhang Wang wrote:
> Hi Srikanth,
>
> In you do not care about the inter
me)
Each process() will do some transformation and do context.forward(...)
Srikanth
ent.truststore.jks
ssl.truststore.password=
Any suggestions on how to debug this?
Srikanth
Hello,
Wanted to check if these JIRAs are on track for 0.10.1.0.
https://issues.apache.org/jira/browse/KAFKA-3478
https://issues.apache.org/jira/browse/KAFKA-3705
Now that 0.10.0.1 is out will the next release be 0.10.1.0 or another bug
fix release?
Srikanth
. Which
is not very efficient. Or may be there is a way around it?
A simple search on RocksDB shows there is a merge operator. That can be of
use here??
Srikanth
On Sun, Sep 11, 2016 at 11:19 PM, Guozhang Wang wrote:
> Hello Ara,
>
> On the processor API, users have the flexible to do micro
e stealing this thread from Ara :-). Lets wait and hear his
reasons.
Srikanth
On Mon, Sep 26, 2016 at 4:59 AM, Michael Noll wrote:
> Ara,
>
> may I ask why you need to use micro-batching in the first place?
>
> Reason why I am asking: Typically, when people talk about micro-batching
ager.close(ProcessorStateManager.java:343)
at
org.apache.kafka.streams.processor.internals.AbstractTask.close(AbstractTask.java:112)
at
org.apache.kafka.streams.processor.internals.StreamTask.close(StreamTask.java:317)
Srikanth
Tnx! Looks like fix is already in for 0.10.1.0
On Tue, Oct 4, 2016 at 6:18 PM, Guozhang Wang wrote:
> Created https://issues.apache.org/jira/browse/KAFKA-4253 for this issue.
>
>
> Guozhang
>
> On Tue, Oct 4, 2016 at 3:08 PM, Guozhang Wang wrote:
>
> > Hello Sri
Hello,
Does kafka stream provide an API to control how internal topics are named?
Right now it uses appId, operator name, etc.
In a shared kafka cluster its common to have naming convention that may
require some prefix/suffix.
Srikanth
That is right Michael. Most teams that use kafka library can adhere to
certain naming convention.
Using streams API will break that.
Srikanth
On Wed, Nov 16, 2016 at 2:32 PM, Michael Noll wrote:
> Srikanth,
>
> no, there's isn't any API to control the naming of internal
Thanks Matthias/Michael/Guozhang!
Using app id may help to some extent. Will have to think & test this
through.
Good to know there will be more direct support for this in the future. May
be it will play well with KIP-37.
Srikanth
On Fri, Nov 18, 2016 at 1:12 PM, Guozhang Wang w
--
> Thanks,
> Kishore.
>
--
Srikanth
, is one planned for 0.11.x? If
so, is 0.11.0.1 or such planned - and when?
* How is 0.10.2.1 in comparison? Why would you recommend me or not to
move to 0.11.0.0 on a practical note?
Thanks a bunch in advance.
-Srikanth
Hi,
Would like to subscribe.
-Srikanth
the kafka config file server.properties at that particular
time also the msgs are lost.
-- So is there any other procedure to update the Zookeeper and kafka
configurations without loosing the Messages.
-- I am trying to find the solution on Google but failed.
Please help
Regards,
Srikanth
We are starting a .Net project using Kafka, I have done a bit of research
and found few repos that have c# clients. but I didn't find any
documentation, how to use those clients programmatically. I was bit
confused how to configure, Consume, Produce, commitoffset in my project.
Can some help me on
Hi,
What is the best recommended replication-factor for a 5 broker kafka?
Can anybody suggest me the good replication factor
Regards,
Srikanth
Hi,
I am new to kafka I am testing all the options in kafka.
I am confused about the re-balancing?
How and where to configure the re-balancing option?
Regards,
Srikanth
ance)?
>
> On Tue, Mar 29, 2016 at 12:49 AM, Srikanth Chandika <
> linosrika...@gmail.com>
> wrote:
>
> > Hi,
> >
> > What is the best recommended replication-factor for a 5 broker kafka?
> > Can anybody suggest me the good replication factor
> >
> > Regards,
> > Srikanth
> >
>
ich is OK because it involves
> moving a lot of data around. Internally at LinkedIn we have some scripts we
> use for more intelligently balancing partitions to assure even balances
> based on a number of criteria. I’m hoping to have more to say about this
> later this week.
>
> -T
Hi,
I started working on Apache Kafka and want to be included in users group.
Please include me.
Thank you.
Best Regards,
Srikanth.
60 matches
Mail list logo
