Re: Kafka Streams Failed to rebalance error

One instance with 10 threads may cause rocksdb issues. What is the RAM you have? Also check CPU wait time. Many rocks db instances on one machine (depends upon number of partitions) may cause lot of disk i/o causing wait times to increase and hence slowing down the message processing causing frequ

What could cause unavailable partitions issue and how to fix it?

In course of our streaming application we discovered that it went into hung state and upon further inspection we found that some of the partitions had no leaders assigned. Here is the description of topic: # bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic new-part-advice-key-tabl

Re: getting intermittent TimeoutException at producer side in streams application

that may indicate that there is a connectivity > issue with the brokers. You would need to correlate the logs etc on the > brokers with the streams logs to try and understand what is happening. > > Thanks, > Damian > > On Sun, 18 Dec 2016 at 07:26 Sachin Mittal wrote: >

Re: getting intermittent TimeoutException at producer side in streams application

changelogs and repartition topics) to 2 or 3 using > StreamsConfig.REPLICATION_FACTOR_CONFIG. > > Thanks > Eno > > > > On 9 Jun 2017, at 20:01, Sachin Mittal wrote: > > > > Hi All, > > We still intermittently get this error. > > > > We had

Re: What could cause unavailable partitions issue and how to fix it?

Folks any input on this? Would help us a lot in case we get into this issue in future as right now only option is to delete the changelog table. On Sat, Jun 10, 2017 at 12:19 AM, Sachin Mittal wrote: > In course of our streaming application we discovered that it went into > hung sta

Do we have to query localWindowStore in same java instance we are creating the store

Hi, I have created a simple window store to count occurrences of a given key. My pipeline is: TimeWindows windows = TimeWindows.of(n).advanceBy(n).until(30n); final StateStoreSupplier supplier = Stores.create("key-table") .withKeys(Serdes.String())

How can we re-key and re-partition records based on that key

Hi, I have a topic of records (K1, V1). This topic is partitioned based on the K1. I use multiple streams instances to process these records based on that key so that each instance process certain partitions only. Now I want to re-key the records in that topic with a format (K2, V2) and push them

Kafka compatibility matrix needed

Hi, I would like some help/information on what client versions are compatible with what broker versions in kafka. Some table like this would be good server client 0.80.9 0.10 0.11 0.8 yes ? ?? 0.9 ?yes ?? 0.10

Re: Kafka compatibility matrix needed

s in the > Kafka cluster before upgrading your clients. Version 0.11.0 brokers support > 0.8.x and newer clients. > > Hope that helps. > > Cheers, > > Michał > > On 18/07/17 08:17, Sachin Mittal wrote: > > Hi, > I would like some help/information on what clie

Re: Kafka compatibility matrix needed

ker versions support all older client versions > > On Tue, Jul 18, 2017 at 10:15 AM, Sachin Mittal > wrote: > > > Hi, > > This gives me some information but still not the complete picture. > > > > It says: > > 0.10.2, Java clients have acquired the

Re: Kafka compatibility matrix needed

s do not work with 0.10.0. We > > upgraded our clients (KafkaSpout and custom consumers) to 0.9.0.1 and > then > > Kafka produce/consume worked fine. > > > > --John > > > > On Tue, Jul 18, 2017 at 6:36 AM, Sachin Mittal > wrote: > > > > > OK. >

Re: Kafka compatibility matrix needed

hat is not supported. > > As usual, you should test this in a non production environment first. Also, > one additional comment inline. > > On Wed, Jul 19, 2017 at 1:19 AM, Sachin Mittal wrote: > > > > I btw have doubt that different versions work on same topic as message &

Can we use builder.stream twice in single application

Hi, I have a multiple high level streams DSL to execute. In first it reads from a source topic processes the data and sends the data to a sink. In second it again reads from same source topic processes the data and sends it to a different topic. For now these two operations are independent. Now

Why does kafka-consumer-groups show the topics written to too in its describe

Hi, I am executing following command bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092 --describe --group new-part-advice It gives output like GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER new-par

Is there a way in increase number of partitions

Hi, I have a topic which has four partitions and data is distributed among those based on a specified key. If I want to increase the number of partitions to six how can I do the same and also making sure that messages for a given key always go to one (specific) partition only. Will the existing m

Re: Why does kafka-consumer-groups show the topics written to too in its describe

. Thanks Sachin On Sun, Aug 6, 2017 at 5:57 PM, Sachin Mittal wrote: > Hi, > I am executing following command > bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server > localhost:9092 --describe --group new-part-advice > > It gives output like > > GROUP

Re: Is there a way in increase number of partitions

n-org. > apache.edgent.function.Function-org.apache.edgent.function.Function- > > > > On 21 August 2017 at 12:02, Sachin Mittal wrote: > > > Hi, > > I have a topic which has four partitions and data is distributed among > > those based on a specified key. > > > > If I want to increa

Kafka streams application failed after a long time due to rocks db errors

Hi, We have been running a clustered kafka streams application and say after 3 months or so of uninterrupted running few threads of couple of instances failed. We checked the logs and we found these two common stack traces pointing to underlying cause of fetch and put operations of rocksdb. Cause

Re: Kafka Streams with embedded RocksDB

Well as far as I remember this was an old issue with 0.10.0.x or something. In 0.10.2.x librocksdbjni dll is part of rocksdbjni-5.0.1.jar so there is no need to build separately for windows. Did something got changed in 0.11.x ? Thanks Sachin On Fri, Oct 6, 2017 at 10:00 PM, Ted Yu wrote: > I

Re: Kafka Streams with embedded RocksDB

I have checked rocksdbjni-5.0.1.jar included in kafka_2.12-0.11.0.0.tgz and librocksdbjni dll is included. I don't think you have to build anything extra fir windows OS. On Sat, Oct 7, 2017 at 11:39 PM, Valentin Forst wrote: > Hi Ted, > > Thank you for your reply... > Yes, I have read the Info

Understanding org.apache.kafka.streams.errors.TopologyBuilderException

Hi, I am getting this exception Exception in thread "main" org.apache.kafka.streams.errors.TopologyBuilderException: Invalid topology building: KTABLE-MAPVALUES-07 and KSTREAM-AGGREGATE-09 are not joinable What I am trying to do is I aggregate a KStream into a KTable of type KTabl

Re: Understanding org.apache.kafka.streams.errors.TopologyBuilderException

I don't think that is the issue. The join api says: public KTable join(KTable other, ValueJoiner joiner) In my case V is Map> V1 is List R is Map> K is String Note the final V and V1 are arrived after doing transformation on original streams So there are intermediate steps like stream.map(new K

Understanding how joins work in Kafka streams

Hi, I needed some light on how joins actually work on continuous stream of data. Say I have 2 topics which I need to join (left join). Data record in each topic is aggregated like (key, value) <=> (string, list) Topic 1 key1: [A01, A02, A03, A04 ..] Key2: [A11, A12, A13, A14 ..] Topic 2 key

Re: Understanding org.apache.kafka.streams.errors.TopologyBuilderException

the topic use in .through() manually with the right > number of partitions before you start you application. > > > -Matthias > > On 10/08/2016 08:54 AM, Sachin Mittal wrote: > > I don't think that is the issue. > > The join api says: > > public KTable join

Re: Understanding how joins work in Kafka streams

it will be called every time a key is modified or it buffers the changes and calls it once in a given time span. Thanks Sachin On Sun, Oct 9, 2016 at 3:07 PM, Eno Thereska wrote: > Hi Sachin, > > Some comments inline: > > > On 9 Oct 2016, at 08:19, Sachin Mittal wrote: &

Re: How to merge two topics in kafka?

You can use topicA.leftJoin (topicB).to (new-topic). You can the consume message from that new topic via second process. Note you need to create all three topics in zookeeper first. On 10 Oct 2016 5:19 a.m., "Ratha v" wrote: Hi all; I have two topics in the broker. Without consuming from one t

Re: How to merge two topics in kafka?

; > Thank you.I'll try the solution. > > But in the highlevel consumer API, topics will be created automatically? > , > > We are not using zookeeper? > > > > On 10 October 2016 at 12:34, Sachin Mittal wrote: > > > >> You can use topicA.leftJoin (top

Hard delete topics

Hi, We are doing some testing and need to frequently wipe out the kafka logs and delete the topic completely. There is no problem starting/stopping zookeeper and server multiple times. So what is the best way of purging a topic and removing its reference from zookeeper entirely. I can physically

Re: Kafka Streaming

You can build librocksdbjni locally to fix it. I did that in my case. It is bit tricky and you need MS visual studio 15. I suggest use the following link: http://mail-archives.apache.org/mod_mbox/kafka-users/201608.mbox/%3CCAHoiPjweo-xSj3TiodcDVf4wNnnJ8u6PcwWDPF7LT5ps%2BxQ3eA%40mail.gmail.com%3E

Understanding zookeper and kafka server failures

Hi, We have following setup. 1. Three instances of zookeeper on three machines. 2. Three instances of kafka server on same three machines. 3. All the topics have replication factor 3. So when we create a topic on any node, i see that it gets replicated on all three instances. I also see that topic

Re: Understanding zookeper and kafka server failures

>> server/broker failure. >> > Again, you shouldn't run zookeeper and kafka on the same server in > production. So the difference is that while kafka is responsible for data, > zookeeper is coordinating tasks between kafka nodes. > > > > On 2016.11.08 17:12, Sachin

Deleting topics in windows - fix estimate or workaround

Hi All, I hope we are well aware of the critical bug in windows where kafka crashes when we delete a topic. This affects other areas too like in stream processing when trying to reset a stream using StreamsResetter. Right now only workaround I have found is to stop zookeeper and kafka server and m

Re: Deleting topics in windows - fix estimate or workaround

you chose to run it on Windows > > > On Sun, Dec 1, 2019 at 2:22 AM Sachin Mittal wrote: > > > Hi All, > > I hope we are well aware of the critical bug in windows where kafka > crashes > > when we delete a topic. This affects other areas too like in stream &g

Re: Deleting topics in windows - fix estimate or workaround

for a contract and demand windows support. > Then magically it happens :) > > On Sun, Dec 1, 2019 at 12:47 PM Sachin Mittal wrote: > > > Hi, > > Firstly development environment is windows and I have been able to run > > almost any big data framework on windows (or

Re: [ANNOUNCE] Apache Kafka 2.2.2

Does this release has fix for that critical windows bug of not able to delete topics? If not then under which release we can expect the same. Thanks and Regards Sachin On Mon, Dec 2, 2019 at 5:55 PM Karolis Pocius wrote: > 2.2.2 is a bugfix release, it contains some of the fixes from 2.3.0/1,

Case of joining multiple streams/tables

Hi, I have checked the documentation and what I see that we can join two streams or tables at a given time. I have a case where I have multiple streams which I need to join based on common key. As of now I am first joining two and the result of that with next and so on. Is there a way or any cas

Re: Case of joining multiple streams/tables

are not fulfilled by the DSL > you can use the Processor API to implement your own join. > > See the following StackOverflow question for more details on joins. > > https://stackoverflow.com/questions/53485632/kafka-streams-implementing-joining-using-the-processor-api > > Bes

How to set concrete names for state stores and internal topics backed by these

Hi, In my application I have names of internal topics like this: ss-session-application-KSTREAM-JOINOTHER-59-store-changelog-0 ss-session-application-KSTREAM-JOINTHIS-49-store-changelog-0 ss-session-application-KSTREAM-OUTEROTHER-50-store-changelog-0 ss-session-application-

Kafka Streams Topology describe format

Hi, I am just posting a section of my topology to basically understand what describe method actually displays. What can we understand just by looking at the topology (like what does --> and <-- arrows represent).

What timestamp is used by streams when doing windowed joins

Hi, I have noticed some issues when doing stream to stream windowed joins. Looks like my joined stream does not include all the records. Say I am doing join like this: stream1.join( stream2, (lv, rv) -> ..., JoinWindows.of(Duration.ofMinutes(5)),

How are custom keys compared during joins

Hi, I am facing some weird problems when joining two streams or a stream/table. The joined stream does not contain all the joined records. Also note that my keys are custom keys for which I have implemented equals and hashcode method Is there something else also I need to do to ensure key1 === key

Re: What timestamp is used by streams when doing windowed joins

t; for writing programs. Both ingestion time and processing time lead to > non-deterministic programs with unclear semantics. That's why we pretty > much stick to event time in the Streams DSL. > > Finally, yeah, if you just want to process records in the same order they > appear

Re: How to set concrete names for state stores and internal topics backed by these

cut" makes me irrationally angry, and I'm hoping we can > find a way to fix it, if we ever change the Materialized builder interface. > > Hope this helps, > -John > > On Fri, Dec 6, 2019, at 11:15, Sachin Mittal wrote: > > Hi, > > In my application I have names

Re: How are custom keys compared during joins

for same key, it won't get joined. So now I have switched back to stream to stream window join. Let me know if there is any other way to handle such a case. Thanks Sachin On Sat, Dec 7, 2019 at 12:08 PM Sachin Mittal wrote: > Hi, > I am facing some weird problems when joining two

Is there a way to prevent duplicate messages to downstream

Hi, I am using streams and I get messages like: (K, V) (A, a), (B, b), (C, c), (A, a), (C, c), (A, a) . I wanted to define a topology which would filter out duplicate messages from upstream. I want to know if this is possible? The code I have written to do this is something like this: source.

Re: Is there a way to prevent duplicate messages to downstream

inc/kafka-streams-examples/blob/5.3.1-post/src/test/java/io/confluent/examples/streams/EventDeduplicationLambdaIntegrationTest.java > . > > > Also, what is the reason for wanting to eliminate duplicates? Do you have > downstream aggregators that you don't want to incorrectly coun

Re: Clickstream partition design question

Its better to have partition key based on some f(user). This way a partition will always have same set of users and any new user would get assigned to one of these partitions. You can probably check https://spark.apache.org/docs/2.2.0/streaming-kafka-0-10-integration.html For kafka to spark integr

Kafka trunk vs master branch

Hello Folks, I just wanted to know what commits goes into what branch. I see trunk branch which seems default and latest. I also see master branch which seems bit behind trunk. I also see different versions branches like 2.2, 2.3 and 2.4 which are also actively updated. I wanted to know when fork

Re: designing a streaming task for count and event time difference

Try something like this: stream .groupBy( (key, value) -> value.userId ) .aggregate( () -> new Session(), (aggKey, newValue, aggValue) -> { aggValue.userId = newValue.userId if (newValue.start) { aggValue.start = newValue.start aggValue.duration = 0

Re: Streams Newbie Question - Deployment and Management of Stream Processors

Hi, The way I have used streams processing in past; use case to process streams is when you have a continuous stream of data which needs to be processed and used by certain applications. Since in kafka streams can be a simple java application, this application can run in its own JVM which is differ

Re: Kafka Streams: out-of-order events when processing multiple topics

I had encountered similar problem and I was using stream to table join. However if a record arrives late in the table, it would not join as per stream to table join semantics. Unfortunately windowing is not supported in stream to table join. I suggest to use stream to stream join with appropriate

Re: Streams Newbie Question - Deployment and Management of Stream Processors

I think literature on confluent/ASF and also the community support here is best to learn about streaming. On Mon, Jan 13, 2020 at 6:47 PM M. Manna wrote: > Hey Sachin, > > On Mon, 13 Jan 2020 at 05:12, Sachin Mittal wrote: > > > Hi, > > The way I have used streams proc

Re: Streams, Kafka windows

You can try to convert the final resultant stream to table. Check this page for more info: https://docs.confluent.io/current/streams/faq.html#how-can-i-convert-a-kstream-to-a-ktable-without-an-aggregation-step This way table would always contain the latest (single) record for a given key. Sachin

Any available cassandra source connector for kafka

Hi, I was looking for any cassandra source connector to read data from cassandra column family and insert the data into kafka topic. Are you folks aware of any community supported version of such a tool. I found one such: https://docs.lenses.io/connectors/source/cassandra.html However I am not a

Kafka connect not loading classes from uber jar

Hi, I am following the documentation under: https://kafka.apache.org/documentation/#connect_user https://docs.confluent.io/current/connect/userguide.html For testing I am using standalone mode. And I am using kafka_2.12-2.3.1. So I have defined: plugin.path=/path/to/plugin/dir In file: config/conn

Re: Kafka connect not loading classes from uber jar

warning and what may be causing it. Thanks Sachin On Thu, Jan 16, 2020 at 2:12 PM Sachin Mittal wrote: > Hi, > I am following the documentation under: > https://kafka.apache.org/documentation/#connect_user > https://docs.confluent.io/current/connect/userguide.html > For testing I am u

Re: KTable Suppress not working

Hi, As far as my understanding goes, aggregated result for a window is not included in next window. Window would stay in state store till it gets deleted based on certain setting however aggregated result for that window will include only the records that occur within the window duration. If you h

Kafka streams DSL going into infinite loop

Hi, In a case I have found that when I define my topology using streams DSL it tends to go into infinite loop. This usually happens if I start my stream and shut it down and restart it again after a while (by that time source topic has moved ahead). Stream processing seems to be stuck in a loop an

Re: Kafka streams DSL going into infinite loop

etween joins, ie, is there multiple > sub-topologies that are connected via repartition topics, or is it one > single sub-topology? > > > -Matthias > > On 1/30/20 10:29 AM, Sachin Mittal wrote: > > Hi, > > In a case I have found that when I define my topology using str

wanted to understand WindowStore#fetchAll(timeFrom, timeTo)

Hi, Say I want to fetch all the keys for store.fetchAll(8, 12) +---+ | key | start time | end time | +---++--+ | A | 10 |20| +---++--+ | A | 15 |25| +---++-

Can we use transform to exchange data between two streams

Hi, I have two streams and I want to enrich stream2 records based off stream1 records. I really cannot join those two streams is because there is no common key between them. Hence only way I can do that is using a timestamp field property. This is how I have built my pipeline. . //create and a

Re: Can we use transform to exchange data between two streams

time-range (since you used fetchAll() > which would return ALL the key-values falling into that range) I think this > would be the way to do it. > > > Guozhang > > > On Wed, Feb 19, 2020 at 10:25 PM Sachin Mittal wrote: > > > Hi, > > I have two streams and

Understanding concept of grace in JoinWindows

Hi, Reading the kafka docs I see that grace period is defined as: the time to admit late-arriving events after the end of the window I however have not understood as when to use it? If I see that some records are arriving after the end of the window and hence not included in the join, should I no

Re: Can we use transform to exchange data between two streams

(streamBB) results are only > needed for populating the store, you can use `process(..)` instead of > `transform(..)` as it would not need a return value since it would be the > end of this branch of the topology. > > > Guozhang > > On Thu, Feb 20, 2020 at 7:32 PM Sachin Mittal w

There seems some issue with javadoc on KStream#transformValues

Hi, The javadoc of this method states: If the return value of ValueTransformer#transform() is null, no records are emitted. However when I check the underlying processor for the same: KStreamT

Re: Understanding concept of grace in JoinWindows

+ join) > -> result > s2 -> k1 (out-of-order record, within the window: insert into store + join) > -> 2 result2 , > > Because we changes the window size, we get 5 result records instead of 2 > (or 1) as in the first two examples. > > Does this make sense? > >

Re: Understanding concept of grace in JoinWindows

on the upstream application that produces the > data. > > Btw: I gave a talk about time semantics at Kafka Summit recently, so > you might want to check out the recording (there will be a follow up > talk at Kafka Summit London in April focusing on time semantics in > Kafka Streams)

Re: Can we use transform to exchange data between two streams

t than relying on the existing in-built > operators, because what you've described here really sounds like a > three-way join to me except for streamB it is not joined by key but by time > window. > > Guozhang > > > > On Fri, Feb 21, 2020 at 8:31 PM Sachin Mittal wrote

Re: How to set the log level of Kafka clients?

I use log4j properties. I think there must be a way to pass log4j settings programmatically as well. If you are using java based producers/consumers then you can set log4j properties and make them available to you classpath before starting those applications. On Sun, Feb 23, 2020 at 9:54 PM Prad

Re: How to set the log level of Kafka clients?

that the log4j > properties file name can be provided in the Java runtime option as - > Dlog4j.configuration=log4propertiesFile. > > Any pointers on whether or how it can be done programmatically would be > greatly helpful. > > Regards, > Pradhan. > > On Sun 23 Fe

Re: KStream.groupByKey().aggregate() is always emitting record, even if nothing changed

Hi, This is really getting interesting. Now if we don't want a record to be emitted downstream only way we can do is via transform or (flatTransform). Since we are now reverting the fix for null record in transformValues and rather change the docs, doesn't this add bit of confusion for users. Conf

Are RocksDBWindowStore windows hopping or sliding?

Hi, So far how I have understood is that when we create a rocksdb window store; we specify a window size and retention period. So windows are created from epoch time based on size, say size if 100 then windows are: [0, 100), [100, 200), [200, 300) ... Windows are retained based on retention perio

Re: KStream.groupByKey().aggregate() is always emitting record, even if nothing changed

o triggering any repartitioning. > > Best, > Bruno > > On Tue, Feb 25, 2020 at 6:43 PM Sachin Mittal wrote: > > > > Hi, > > This is really getting interesting. > > Now if we don't want a record to be emitted downstream only way we can do > > is via transfo

Re: Are RocksDBWindowStore windows hopping or sliding?

on for hopping windows > (that we use for tumbling windows, too): > > https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamWindowAggregate.java > > > -Matthias > > On 2/26/20 9:47 PM, Sachin Mittal wrote: > > Hi, &

Re: Passing states stores around

Well I can suggest is that you can access state store via processor's context. Once you have the reference of the state store it can be passed to different classes via the same reference. On Tue, Mar 10, 2020 at 8:57 AM Navneeth Krishnan wrote: > Hi John, > > I'm using PAPI to create my topolog

Re: Kafka JMX monitoring

How about jmxterm. https://cwiki.apache.org/confluence/display/KAFKA/jmxterm+quickstart I have found this very helpful. On Thu, Mar 19, 2020 at 3:51 PM Gioacchino Vino wrote: > Hi, > > > I would like to share my experience with you. > > Even if I found out that prometheus is the a simple and fa

Re: Are RocksDBWindowStore windows hopping or sliding?

exact same manner for hopping window > (compare the code I shared in the last reply). Even if windows are > overlapping, the grouping-key+window-start-timestamp is a unique > primary key for each window. > > - -Matthias > > On 2/27/20 9:26 AM, Sachin Mittal wrote: > > Hi, Yes

Re: Get after put in stateStore returns null

You can try to check the file which rocksdb creates for the state store to see if value is stored for the key. What I suspect is that get may be getting called before put in your topology. Try retrieving the value from state store further downstream. Thanks Sachin On Thu, 2 Apr 2020, 02:35 Jan

How to add partitions to an existing kafka topic

Hi, We have a kafka streams application which runs multiple instances and consumes from a source topic. Producers produces keyed messages to this source topic. Keyed messages are events from different sources and each source has a unique key. So what essentially happens is that messages from parti

Re: How to add partitions to an existing kafka topic

bility to consistently route the > > same key data, which is not resolved today. > > > > Boyang > > > > On Wed, Apr 15, 2020 at 7:23 AM Sachin Mittal > wrote: > > > > > Hi, > > > We have a kafka streams application which runs multiple ins

Re: best approach to rotate the logs and archive on Kubernetes

I think if you just log to the console, kubernetes will manage the log rotation for you. https://kubernetes.io/docs/concepts/cluster-administration/logging/ You can use "kubectl logs" command to fetch the logs or use some logging agent to move the logs centrally. On Tue, Aug 4, 2020 at 8:15 PM Sri

How do we usually handle Node disconnected issue for kafka producer

Hi, I am using a Kafka producer java client by vert.x framework. https://vertx.io/docs/apidocs/io/vertx/kafka/client/producer/KafkaProducer.html There is a producer setting in kafka: connections.max.idle.ms = 54 So if there are no records to produce then after 9 minutes I get this in my logs:

Re: How do we usually handle Node disconnected issue for kafka producer

, Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Sachin, > > Why do you want to change the default settings? If the connection is open > and unused, > then it is fair to close the connection after the timeout and reopen it > when required. > > On Fri,

Kafka streaming changelog topic max.message.bytes exception

Hi, We are using aggregation by key on a kstream to create a ktable. As I read from https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams%3A+Internal+Data+Management it creates an internal changelog topic. However over the time the streaming application is run message size increases and

Re: Kafka streaming changelog topic max.message.bytes exception

at you mean by "message size increases"? Are messages > going to the changelog topic increasing in size? Or is the changelog topic > getting full? > > Thanks > Eno > > > On 8 Nov 2016, at 16:49, Sachin Mittal wrote: > > > > Hi, > > We are using aggr

Re: Kafka streaming changelog topic max.message.bytes exception

n Tue, Nov 8, 2016 at 10:00 AM, Sachin Mittal wrote: > > > Message size itself increases over the time. > > > > Message is something like > > key=[list on objects] > > > > This increases with time and then at a point kafka is not able to add any > >

Understanding the topology of high level kafka stream

Hi, I had some basic questions on sequence of tasks for streaming application restart in case of failure or otherwise. Say my stream is structured this way source-topic branched into 2 kstreams source-topic-1 source-topic-2 each mapped to 2 new kstreams (new key,value pairs) backed

Re: Understanding the topology of high level kafka stream

no > different than if you had left the application running (i.e., no restart). > > Thanks > Eno > > > > On 9 Nov 2016, at 13:59, Sachin Mittal wrote: > > > > Hi, > > I had some basic questions on sequence of tasks for streaming application > > restart

Re: Kafka streaming changelog topic max.message.bytes exception

be to precreate all internal topics in > > Kafka, and only after that start the Kafka Streams application. > > This would require you knowing the internal name of the topics (in > > this case you probably already know it, but I agree that in general > > this is a bit cumbersome).

Re: Understanding the topology of high level kafka stream

s still semantically meaningful and compatible if > you start to modify offsets...) > > > Hope this helps. > > - -Matthias > > On 11/9/16 7:29 AM, Sachin Mittal wrote: > > Hi, What happens when the message itself is purged by kafka via > > retention time setting

Re: Understanding the topology of high level kafka stream

t; > > - -Matthias > > On 11/10/16 3:48 AM, Sachin Mittal wrote: > > Hi, The reset tool looks like a great feature. > > > > So following this link > > https://www.confluent.io/blog/data-reprocessing-with-kafka-streams-res > etting-a-streams-application/ >

Re: Kafka streaming changelog topic max.message.bytes exception

lue record it there is not update > for it for some time, you can combine compaction with retention. You > need to alter the configuration of the changelog topics and set > > cleanup.policy=compact,delete > > Than, retention.ms will be applied to the changelog, too. > > > - -M

Kafka windowed table not aggregating correctly

Hi, I have a simple pipeline stream.aggregateByKey(new Initializer() { public List apply() { return new List } }, new Aggregator() { public List apply(key, value, list) { list.add(value) return list } }, keysSerde, valuesSerde, "table") So this basically aggregates list

How to get and reset Kafka stream application's offsets

Hi I am running a streaming application with streamsProps.put(StreamsConfig.APPLICATION_ID_CONFIG, "test"); How do I find out the offsets for each of the source, intermediate and internal topics associated with this application. And how can I reset them to some specific value via shell of otherwi

Re: Kafka windowed table not aggregating correctly

issues in application reset. Thanks Sachin On Fri, Nov 18, 2016 at 11:30 PM, Guozhang Wang wrote: > Hello Sachin, > > Which version of Kafka are you using for this application? > > > Guozhang > > > On Tue, Nov 15, 2016 at 9:52 AM, Sachin Mittal wrote: > > &

Re: How to get and reset Kafka stream application's offsets

://www.confluent.io/blog/data-reprocessing-with- > kafka-streams-resetting-a-streams-application/> > > However, there is no option currently for resetting the offset to an > arbitrary offset. > > Thanks > Eno > > > On 21 Nov 2016, at 10:37, Sachin Mittal wrote: > &

Re: Kafka windowed table not aggregating correctly

ist all is working fine. Anyway any ideas here would be appreciated, meanwhile let me see what is the best java practice here. Thanks Sachin On Mon, Nov 21, 2016 at 8:29 PM, Michael Noll wrote: > On Mon, Nov 21, 2016 at 1:06 PM, Sachin Mittal wrote: > > > I am using kafka_2.10-0.10.0.

Re: Delete "kafka-logs"

Check http://kafka.apache.org/documentation.html#brokerconfigs log.retention.bytes log.retention.ms log.segment.bytes delete.topic.enable On Tue, Nov 22, 2016 at 1:36 PM, Eranga Heshan wrote: > Hi all, > > I want to keep the size of kafka-logs reduced as much as possible. To do it > I need to

Re: Delete "kafka-logs"

2%202087> > Email: era...@wso2.com > <https://www.facebook.com/erangaheshan> <https://twitter.com/ > erangaheshan> ><https://www.linkedin.com/in/erangaheshan> > > On Tue, Nov 22, 2016 at 1:47 PM, Sachin Mittal wrote: > > > Check &g

Re: Delete "kafka-logs"

era...@wso2.com > <https://www.facebook.com/erangaheshan> <https://twitter.com/ > erangaheshan> ><https://www.linkedin.com/in/erangaheshan> > > On Tue, Nov 22, 2016 at 2:15 PM, Sachin Mittal wrote: > > > Set some value to log.retention.m

  1   2   3   >