Re: Kafka InvalidStateStoreException

2017-09-15 Thread Ted Yu
bq. at com.mytest.csd.kafka.KafkaDeduplicator.getQuotaStore(KafkaDe duplicator.java:147) Can you show us relevant code snippet for the above method ? On Fri, Sep 15, 2017 at 2:20 AM, Jari Väimölä wrote: > Hello all, > I have an apache kafka stream application running

Re: Kafka Streams application Unable to Horizontally scale and the application on other instances refusing to start.

2017-09-15 Thread Ted Yu
broker logs ? I haven't looked at the broker >> side since my spark application processing from the same topic with a >> different group id is able to process well. >> >> On Fri, Sep 15, 2017 at 3:30 PM, Ted Yu <yuzhih...@gmail.com> wrote: >> >>> Is

Re: Kafka MirrorMaker - target or source datacenter deployment

2017-09-14 Thread Ted Yu
Wouldn't KAFKA-5494 make remote produce more reliable? Original message From: Todd Palino Date: 9/14/17 6:53 PM (GMT-08:00) To: users@kafka.apache.org Subject: Re: Kafka MirrorMaker - target or source datacenter deployment Always in the target datacenter.

Re: Kafka Streams application Unable to Horizontally scale and the application on other instances refusing to start.

2017-09-15 Thread Ted Yu
ere getting closed and started which might >> have >> > resulted in the "CommitFailedException due to non avialability of >> > processing processors. >> > >> > After some time the issue got propagated to other servers, I have >> attached &

Re: Kafka Streams application Unable to Horizontally scale and the application on other instances refusing to start.

2017-09-16 Thread Ted Yu
Hi, Were you using 0.11.0.0 ? I ask this because some related fixes, KAFKA-5167 and KAFKA-5152, are only in 0.11.0.1 Mind trying out 0.11.0.1 release and see whether the problem persists ? On Fri, Sep 15, 2017 at 12:52 PM, Ted Yu <yuzhih...@gmail.com> wrote: > bq. 1)

Re: Kafka Streams application Unable to Horizontally scale and the application on other instances refusing to start.

2017-09-16 Thread Ted Yu
ere any upgrade which needs to be > done on kafka broker side as well ? > > On Sat, Sep 16, 2017 at 5:18 AM, Ted Yu <yuzhih...@gmail.com> wrote: > > > Hi, > > Were you using 0.11.0.0 ? > > > > I ask this because some related fixes, KAFKA-5167 and KAFKA-5152, ar

Re: Improving Kafka State Store performance

2017-09-16 Thread Ted Yu
Looking at the calculation of totalTimeTakenToStoreRecords, it covers store.put() call. Can you tell us more about what the put() does ? Does it involve external key value store ? Are you using 0.11.0.0 ? Thanks On Sat, Sep 16, 2017 at 6:14 AM, dev loper wrote: > Hi Kafka

Re: Improving Kafka State Store performance

2017-09-18 Thread Ted Yu
We're using rocksdb 5.3.6 It would make more sense to perform next round of experiment using rocksdb 5.7.3 which is latest. Cheers On Mon, Sep 18, 2017 at 5:00 AM, Bill Bejeck wrote: > I'm following up from your other thread as well here. Thanks for the info > above, that

Re: Kafka Connect Sink Connector for multiple JDBC sinks

2017-09-16 Thread Ted Yu
Have you looked at https://github.com/confluentinc/kafka-connect-jdbc ? On Sat, Sep 16, 2017 at 1:39 PM, M. Manna wrote: > Sure. But all these are not available via Kafka open source (requires > manual coding), correct? Only Confluence seems to provide some > off-the-shelf

Re: Kafka Streams application Unable to Horizontally scale and the application on other instances refusing to start.

2017-09-16 Thread Ted Yu
ORDS_CONFIG >> > =6 and the processors were getting closed and started which might >> have >> > resulted in the "CommitFailedException due to non avialability of >> > processing processors. >> > >> > After some tim

Re: Data loss while upgrading confluent 3.0.0 kafka cluster to confluent 3.2.2

2017-09-19 Thread Ted Yu
Looks like the screen shots didn't come through.  Consider pasting the text. Thanks Original message From: Yogesh Sangvikar Date: 9/19/17 4:33 AM (GMT-08:00) To: users@kafka.apache.org Cc: Sumit Arora , Bharathreddy

Re: Hi

2017-09-21 Thread Ted Yu
Please follow instructions on http://kafka.apache.org/contact On Thu, Sep 21, 2017 at 1:30 PM, Daniele Ascione wrote: > hi, I would like to subscribe >

Re: Kafka Internals Video/Blog

2017-09-20 Thread Ted Yu
Kafka has been evolving rapidly. Is there area(s) which you're particularly interested ? You can find all the relevant KIPs under: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals Normally KIPs contain fair amount of technical details. >From each KIP, you can find

Re: Kafka Streams application Unable to Horizontally scale and the application on other instances refusing to start.

2017-09-17 Thread Ted Yu
Ram, Storage : EBS > > Kafka Streams Instance : 3 Kafka Streams Application Instances (Current > CPU Usage 8%- 24%) > > Instance Type : AWS M4 Large > Machine Configuration : 2 VCPU;s, 8gb Ram, Storage : EBS (Dedicated > EBS bandwidth 450 mbps) > Thanks >

Re: Kafka MirrorMaker - exit on send failure?

2017-09-14 Thread Ted Yu
>From MirrorMaker.scala : // Defaults to no data loss settings. maybeSetDefaultProperty(producerProps, ProducerConfig.RETRIES_CONFIG, Int.MaxValue.toString) maybeSetDefaultProperty(producerProps, ProducerConfig.MAX_BLOCK_MS_CONFIG, Long.MaxValue.toString) I think the settings

Re: Kafka + ACL Topic based security ,Spring Boot

2017-09-22 Thread Ted Yu
Attachment didn't come thru. Have you read ? https://cwiki.apache.org/confluence/display/KAFKA/Security especially the https://cwiki.apache.org/confluence/display/KAFKA/Security#Security-ImplementingthePermissionManager section ? On Fri, Sep 22, 2017 at 2:45 AM, Pooppillikudiyil, Joby <

Re: Kafka ACL - Access Denied while publishing the message

2017-10-04 Thread Ted Yu
>From the example off: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Authorization+Command+Line+Interface it seems following 'User:', the formation is te...@example.com Can you double check ? On Wed, Oct 4, 2017 at 8:55 PM, Awadhesh Gupta wrote: > Hi, > > I

Re: Kafka Streams with embedded RocksDB

2017-10-06 Thread Ted Yu
I assume you have read https://github.com/facebook/rocksdb/wiki/Building-on-Windows Please also see https://github.com/facebook/rocksdb/issues/2531 BTW your question should be directed to rocksdb forum. On Fri, Oct 6, 2017 at 6:39 AM, Valentin Forst wrote: > Hi there, > >

Re: kafka broker loosing offsets?

2017-10-06 Thread Ted Yu
gt; > >: > > > > > > > > > Reprocessing same events again - is fine for us (idempotent). While > > > > loosing > > > > > data is more critical. > > > > > > > > > > What are reasons of such behaviour? Consumers

Re: kafka broker loosing offsets?

2017-10-06 Thread Ted Yu
tion to find out what happens. Also in general with current > semantics of offset reset policy IMO using anything but none is not really > an option unless it is ok for consumer to loose some data (latest) or > reprocess it second time (earliest). > > fre 6 okt. 2017 kl. 17:44 skrev

Re: __consumers topic is overloaded

2017-10-06 Thread Ted Yu
Graph image didn't come through. Consider using third party site for hosting image. On Fri, Oct 6, 2017 at 6:46 AM, Alexander Petrovsky wrote: > Hello! > > I observe the follow strange behavior in my kafka graphs: > > > As you can see, the topic __consumer_offsets have very

Re: kafka broker loosing offsets?

2017-10-06 Thread Ted Yu
tes (336 hrs)? > > > On Fri, Oct 6, 2017 at 8:58 PM, Dmitriy Vsekhvalnov < > dvsekhval...@gmail.com> > wrote: > > > Hi Ted, > > > > Broker: v0.11.0.0 > > > > Consumer: > > kafka-clients v0.11.0.0 > > auto.offset.reset = earliest >

Re: kafka broker loosing offsets?

2017-10-06 Thread Ted Yu
What's the value for auto.offset.reset ? Which release are you using ? Cheers On Fri, Oct 6, 2017 at 7:52 AM, Dmitriy Vsekhvalnov wrote: > Hi all, > > we several time faced situation where consumer-group started to re-consume > old events from beginning. Here is

Re: NPE on ConsumerRecords$ConcatenatedIterable$1.makeNext() while iterating records

2017-10-08 Thread Ted Yu
Was there any consumer interceptor involved ? Cheers On Sun, Oct 8, 2017 at 6:29 AM, Michael Keinan wrote: > Hi > Using Kafka 0.10.2.0 > I get a NPE while iterating the records after polling them using poll > method. > - Any idea where does it come from ? > - How can I

Re: Kafka stream api to consume json values using kafka connect json deserializer

2017-10-04 Thread Ted Yu
Have you taken a look at streams/examples/src/main/java/org/apache/kafka/streams/examples/pageview/PageViewUntypedDemo.java ? final Deserializer jsonDeserializer = new JsonDeserializer(); final Serde jsonSerde = Serdes.serdeFrom(jsonSerializer, jsonDeserializer); final

Re: Store - RocksDB - aging data off

2017-10-05 Thread Ted Yu
Past thread related to TTL: http://search-hadoop.com/m/Kafka/uyzND1RLg4VOJ84U?subj=Re+Streams+TTLCacheStore On Thu, Oct 5, 2017 at 9:54 AM, Ted Yu <yuzhih...@gmail.com> wrote: > Looking at > streams/src/main/java/org/apache/kafka/streams/state/internals/Rocks

Re: Store - RocksDB - aging data off

2017-10-05 Thread Ted Yu
Looking at streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java : throw new UnsupportedOperationException("Change log is not supported for store " + this.name + " since it is TTL based."); // TODO: support TTL with change log?

Re: Kafka connect rebalance problem

2017-10-17 Thread Ted Yu
bq. There are 7 connectors configured with each connector configured with 8 tasks (Averaging about 4 tasks per connector) Pardon. I don't quite understand the above setup. Can you describe in more detail ? Which version of connector are you using ? Cheers On Mon, Oct 16, 2017 at 11:30 PM,

Re: Kafka 0.9.0.1 Direct buffer memory OOM

2017-10-17 Thread Ted Yu
Can you take a look at KAFKA-5470 ? On Tue, Oct 17, 2017 at 2:32 AM, 杨文 wrote: > Hi Kafka Users, > We are using kafka 0.9.0.1 and frequently we have seen below exception > which causes the broker > to die.We even increased the MaxDirectMemory to 1G but still see this. >

Re: when will zk path "/kafka/brokers/topics/__consumer_offsets/partitions/{number}/state" be updated

2017-09-08 Thread Ted Yu
The logs didn't go through. Consider using pastebin. For steps 1 and 2, it seems you used some image which didn't go through. Please use pastebin instead. Where did you get the docker image ? Cheers On Fri, Sep 8, 2017 at 4:42 AM, Nick <394299...@qq.com> wrote: > Dears, > > May I seek your

Re: To Know about Kafka and it's usage

2017-09-28 Thread Ted Yu
Parth : bq. an application which could take some real time data Can you be a bit more specific on what your goal is ? It would help narrow down the choices you have. Cheers On Thu, Sep 28, 2017 at 8:01 AM, David Garcia wrote: > If you’re on the AWS bandwagon, you can

Re: Is anyone using MX4J loader?

2017-09-29 Thread Ted Yu
Looks like Ralph logged KAFKA-4946 for this already. On Fri, Sep 29, 2017 at 12:40 AM, Dong Lin wrote: > Hi Kafka users, > > I am wondering if anyone is currently using feature from MX4J loader. This > feature is currently enabled by default. But if kafka_mx4jenable is >

Re: subscription

2017-09-29 Thread Ted Yu
See instruction at https://kafka.apache.org/contact On Fri, Sep 29, 2017 at 7:01 AM, Alex.Chen wrote: > subscription >

Re: windowed store excessive memory consumption

2017-09-27 Thread Ted Yu
Have you seen this comment ? https://issues.apache.org/jira/browse/KAFKA-5122?focusedCommentId=15984467=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15984467 On Wed, Sep 27, 2017 at 12:44 PM, Stas Chizhov wrote: > Hi, > > I am running a simple

Re: kafka-console-producer.sh --security-protocol

2017-10-03 Thread Ted Yu
I think in producer.properties you should use: security.protocol=SASL_PLAINTEXT FYI On Tue, Oct 3, 2017 at 7:17 AM, Pekka Sarnila wrote: > Hi, > > kafka_2.11-0.11.0.0 > > If I try to give --security-protocol xyz (xyz any value e.g. > SASL_PLAINTEXT, PLAINTEXTSASL, SASL_SSL) I

Re: Kafka Streams Avro SerDe version/id caching

2017-10-03 Thread Ted Yu
I did a quick search in the code base - there doesn't seem to be caching as you described. On Tue, Oct 3, 2017 at 6:36 AM, Kristopher Kane wrote: > If using a Byte SerDe and schema registry in the consumer configs of a > Kafka streams application, does it cache the Avro

Re: 2 broker kafka cluster in production

2017-09-25 Thread Ted Yu
bq. 1 topic with replication factor 1 There is no fault tolerance for the above setup. Related please read KIP-113 . Cheers On Mon, Sep 25, 2017 at 11:56 AM, Anu P wrote: > Hi All, > > I would like to request inputs - pros and cons of setting up a 2 broker > Kafka

Re: Fetching topic metadata issue

2017-09-04 Thread Ted Yu
Please give us more information: release of Kafka Did your consumer get any error ? Have you inspected broker log(s) ? Cheers On Sun, Sep 3, 2017 at 11:08 PM, Sagar Nadagoud < sagar.nadag...@wildjasmine.com> wrote: > Hi, > > Please help me in this issue i am unable to read the message from

Re: Kafka cluster Error

2017-10-10 Thread Ted Yu
Can you check the health of zookeeper quorum ? Cheers On Tue, Oct 10, 2017 at 3:35 PM, Kannappan, Saravanan (Contractor) < saravanan_kannap...@comcast.com> wrote: > Hello, Someone can you help me kafka server not starting after rebooting , > the below is the error message > > [2017-10-10

Re: Regarding kafka-manager topics parameters

2017-10-18 Thread Ted Yu
Images didn't come thru. Consider using third party website. On Tue, Oct 17, 2017 at 9:36 PM, Pavan Patani wrote: > Hello, > > Previously I was using old version of Kafka-manager and it was showing > "Producer Message/Sec and Summed Recent Offsets" parameters in

Re: Kafka FileStreamSinkConnector handling of bad messages

2017-10-18 Thread Ted Yu
Considering Ewen's response, you can open a JIRA for applying the suggestion toward FileStreamSinkConnector. Cheers On Wed, Oct 18, 2017 at 10:39 AM, Marina Popova wrote: > Hi, > I wanted to give this question a second try as I feel it is very > important to

Re: Kafka problem

2017-11-22 Thread Ted Yu
Can you provide more information (such as pastebin of relevant logs) ? Cheers On Wed, Nov 22, 2017 at 1:55 AM, Linux实训项目 wrote: > Hi: >Kafka node zookeeper is lost after a period of time.What causes the > node to be lost in zookeeper? > > >environment: >

Re: Too many open files in kafka 0.9

2017-11-29 Thread Ted Yu
There is KAFKA-3317 which is still open. Have you seen this ? http://search-hadoop.com/m/Kafka/uyzND1KvOlt1p5UcE?subj=Re+Brokers+is+down+by+java+io+IOException+Too+many+open+files+ On Wed, Nov 29, 2017 at 8:55 AM, REYMOND Jean-max (BPCE-IT - SYNCHRONE TECHNOLOGIES)

Re: StackOverflowError in kafka-coordinator-heartbeat-thread with Kafka 1.0

2017-12-14 Thread Ted Yu
Interesting. Looks like disconnection resulted in the stack overflow. I think the following would fix the overflow: https://pastebin.com/Pm5g5V2L On Thu, Dec 14, 2017 at 7:40 AM, Jörg Heinicke wrote: > > Hi everyone, > > We recently switched to Kafka 1.0 and are facing

Re: caching permantly disabled in Kafka Streams

2017-12-14 Thread Ted Yu
In StreamsConfig.java , CACHE_MAX_BYTES_BUFFERING_CONFIG is defined as: .define(CACHE_MAX_BYTES_BUFFERING_CONFIG, Type.LONG, 10 * 1024 * 1024L, I think using numeral should be accepted (as shown by the Demo.java classes). On Thu, Dec 14,

Re: Kafka isync replica

2017-12-14 Thread Ted Yu
Can you look at the log from controller to see if there is some clue w.r.t. partition 82 ? Was unclean leader election enabled ? BTW which release of Kafka are you using ? Cheers On Thu, Dec 14, 2017 at 11:49 AM, Tarun Garg wrote: > I checked log.dir of the all nodes and

Re: Producer is blocking at second commitTransaction

2017-12-18 Thread Ted Yu
Can you capture stack trace on the broker and pastebin it ? Broker log may also provide some clue. Thanks On Mon, Dec 18, 2017 at 4:46 AM, HKT wrote: > Hello, > > I was testing the transactional message on kafka. > but I get a problem. > the producer always blocking at

Re: Need help in storing key along with value of _schemas topic using s3 sink connector.

2017-12-13 Thread Ted Yu
Have you checked s3 connector issue listing ? Cheers On Wed, Dec 13, 2017 at 2:38 AM, Pratik Shah wrote: > Hi All, > I am using kafka s3 sink connector ( version 4.0) to backup the > _schemas topic for its backup. > The issue is that the content stored

Re: Class Files\OpenLink not found

2017-12-13 Thread Ted Yu
Which release of Kafka did you download ? The error seems to be (in English): unable to find or load the main class I searched Kafka source code but didn't find reference to OpenLink. Can you give the complete error ? Thanks On Wed, Dec 13, 2017 at 12:55 AM, Thomas JASSEM

Re: ConcurrentModificationException

2017-12-12 Thread Ted Yu
>From the stack trace, it seems you hit KAFKA-6349 (where Damian provided a PR). FYI On Mon, Dec 11, 2017 at 11:29 PM, Mr.Wang <1282183...@qq.com> wrote: > Hi~ > We found this error when we read kafka data using storm: > java.util.ConcurrentModificationException at >

Re: Producer is blocking at second commitTransaction

2017-12-18 Thread Ted Yu
ing > transaction metadata from __transaction_state-15 > (kafka.coordinator.transaction.TransactionStateManager) > [2017-12-19 08:26:08,471] INFO [Transaction State Manager 0]: Loading > transaction metadata from __transaction_state-18 > (kafka.coordinator.transaction.Transacti

Re: Time-Based Index for Consuming Messages Up to Certain Timestamp

2017-11-21 Thread Ted Yu
bq. an older timestamp that allowed I guess you meant 'than allowed' Cheers On Tue, Nov 21, 2017 at 2:57 PM, Matthias J. Sax wrote: > This is possible, but I think you don't need the time-based index for it :) > > You will just buffer up all messages for a 5 minute

Re: Authentication using ssl in kafka

2017-11-01 Thread Ted Yu
Neither the error nor properties files went through. Can you use pastebin or some similar site ? On Wed, Nov 1, 2017 at 5:31 AM, mascarenhas, jewel < mascarenhas.je...@atos.net> wrote: > Hi, > I am trying to configure apache kafka version 0.11.0.1 with ssl, but am > faing the following error

Re: documentation link broken

2017-11-02 Thread Ted Yu
See: https://github.com/apache/kafka/blob/1.0/clients/src/main/java/org/apache/kafka/clients/admin/AdminClient.java On Thu, Nov 2, 2017 at 5:51 AM, diane wrote: > Hi > > I was trying to look at the documentation for the AdminClient API , but > the link from page >

Re: Getting rid of Attempting to send response via channel for which there is no open connection

2017-11-02 Thread Ted Yu
Have you seen this thread ? http://search-hadoop.com/m/Kafka/uyzND1ppGvmNscWc?subj=Re+Data+loss+while+upgrading+confluent+3+0+0+kafka+cluster+to+confluent+3+2+2 On Thu, Nov 2, 2017 at 1:46 PM, Karim Lamouri wrote: > Hi, > > I wanted to know in which cases those warning arise

Re: Using Kafka to access APIs.

2017-11-07 Thread Ted Yu
I think how to use GDAX's API is orthogonal to using Kafka. Kafka client has support for Java and Python. On Tue, Nov 7, 2017 at 12:31 PM, Taha Arif wrote: > Hello, > > > I want to build a project that accesses the Gdax websocket in a real time > stream, and push that

Re: GlobalKTable never finishes restoring

2017-11-07 Thread Ted Yu
Alex: In the future, please use pastebin if the log is not too large. When people find this thread in mailing list archive, the attachment wouldn't be there. Thanks On Tue, Nov 7, 2017 at 8:32 AM, Matthias J. Sax wrote: > Alex, > > I am not sure, but maybe it's a bug. I

Re: Migrating from log4j 1 to log4j 2

2017-11-09 Thread Ted Yu
PropertyConfigurator is used here: https://github.com/apache/kafka/blob/trunk/tools/src/main/java/org/apache/kafka/tools/VerifiableLog4jAppender.java#L233 On Thu, Nov 9, 2017 at 2:17 PM, Arunkumar wrote: > Hi All > We have a requirement to migrate log4J 1.x to

Re: Kafka JVM heap limit

2017-11-08 Thread Ted Yu
Did you use G1GC ? Thanks Original message From: John Yost Date: 11/8/17 5:48 AM (GMT-08:00) To: users@kafka.apache.org Cc: ja...@scholz.cz Subject: Re: Kafka JVM heap limit In addition, in my experience, a memory heap > 8 GB leads to long GC pauses

Re: Kafka Monitoring..

2017-11-08 Thread Ted Yu
Have you seen this thread ? http://search-hadoop.com/m/Kafka/uyzND1Q6wyNBj42g?subj=Kafka+Monitoring On Wed, Nov 8, 2017 at 5:10 PM, chidigam . wrote: > Hi All, > What is the simplest way of monitoring the metrics in kaka brokers? > Is there any opensource available? >

Re: Client Offset Deleted by Broker without cause

2017-11-01 Thread Ted Yu
bq. attached screenshots from the log viewer The screenshots didn't go through. Consider using 3rd party site. On Wed, Nov 1, 2017 at 9:18 AM, Elmar Weber wrote: > Hello, > > I had this morning the issue that a client offset got deleted from a > broker as it seems. > >

Re: Offset commit for partitions not owned by consumer

2017-11-03 Thread Ted Yu
Can you pastebin relevant logs from client and broker ? Thanks On Fri, Nov 3, 2017 at 1:37 PM, Manan G wrote: > Hello, > > I am using 0.11.0.0 version of Kakfa broker and Java client library. My > consumer code tracks offsets for each assigned partition and at some time >

Re: Offset commit for partitions not owned by consumer

2017-11-07 Thread Ted Yu
tebin.com/PuQhud91 > > Consumer 2 logs: https://pastebin.com/yfJDSGPA > > > > server.log: https://pastebin.com/QKpk0zLn > > controller.log: https://pastebin.com/9T0niwEw > > state-change.log: https://pastebin.com/nrftHPC9 > > > > > > On Fri, Nov 3, 2017 at 1:5

Re: Getting install error ...

2017-12-11 Thread Ted Yu
In the guide, note the following (see bin/windows): change the script extension to .bat On Mon, Dec 11, 2017 at 2:57 AM, kish babu wrote: > Hi, > I am trying to setup Kafka on my windows box which has no cygwin. > > I am following instructions from > >

Re: Servers Getting disconnected

2017-10-30 Thread Ted Yu
Please take a look at KAFKA-4477 On Mon, Oct 30, 2017 at 2:06 AM, Yuanjia wrote: > Hi, > My cluster version is 0.10.0.0, 10 nodes. One server has many error > log, like that > java.io.IOException: Connection to 6 was disconnected before the response > was read >

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

2017-10-25 Thread Ted Yu
Have you seen the email a moment ago from Onur which uses the same KIP number ? Looks like there was race condition in modifying wiki. Please consider bumping the KIP number. Thanks On Wed, Oct 25, 2017 at 4:14 PM, Jan Filipiak wrote: > Hello Kafka-users, > > I want

Re: Consumer poll returning 0 results

2017-10-25 Thread Ted Yu
Can you provide a bit more information ? Release of Kafka Java / Scala version Thanks On Wed, Oct 25, 2017 at 6:40 PM, Susheel Kumar wrote: > Hello Kafka Users, > > I am trying to run below sample code mentioned in Kafka documentation under > Automatic Offset Committing

Re: Failure to reset consumer offsets for specific topics

2017-10-26 Thread Ted Yu
08978014698 offset: 254410828 > > > timestamp: 1508978014959 offset: 254410834 > > > timestamp: 1508978015677 offset: 254410854 > > > timestamp: 1508978016980 offset: 254410870 > > > timestamp: 1508978017212 offset: 254410883 > > > >

Re: Failure to reset consumer offsets for specific topics

2017-10-25 Thread Ted Yu
then tried to manually reset the consumer group's offset to a few > minutes before I restarted the broker, only to discover this strange > behavior where no matter which datetime value I provided, it kept resetting > to the latest offset. > > > On 25 October 2017 at 22:48, Ted Y

Re: Failure to reset consumer offsets for specific topics

2017-10-25 Thread Ted Yu
Clarification: my most recent reply was w.r.t. the strange situation Dan described, not the offset resetting. On Wed, Oct 25, 2017 at 1:24 PM, Ted Yu <yuzhih...@gmail.com> wrote: > I wonder if you have hit KAFKA-5600. > > Is it possible that you try out 0.11.0.1 ? > > Than

Re: Failure to reset consumer offsets for specific topics

2017-10-25 Thread Ted Yu
Do you mind providing a bit more information ? Release of Kafka you use Any difference between data1_log and the other, normal topic ? Probably check the broker log where data1_log is hosted - see if there is some clue. Thanks On Wed, Oct 25, 2017 at 12:11 PM, Dan Markhasin

Re: Kafka source code - Build Error

2017-10-24 Thread Ted Yu
This should fix what you observed: https://github.com/apache/kafka/pull/4127 On Tue, Oct 24, 2017 at 12:21 PM, Vishal Shukla wrote: > Hi All, > I took latest source code from kafka git repo and tried to setup my local > env. in Eclipse. > I am getting a compilation error in

Re: Kafka source code - Build Error

2017-10-24 Thread Ted Yu
JDK 1.8.0_91 Cheers On Tue, Oct 24, 2017 at 1:23 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Hmm.. which Java version were you using? > > > Guozhang > > On Tue, Oct 24, 2017 at 1:09 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > This should fix what y

Re: Kafka Streams 0.11.0.1 - Task Assignment Stickiness Behavior

2017-10-24 Thread Ted Yu
Eric: I wonder if it is possible to load up 1.0.0 RC3 on test cluster and see what the new behavior is ? Thanks On Tue, Oct 24, 2017 at 5:41 PM, Eric Lalonde wrote: > > >>> Could it be, that the first KafkaStreams instance was still in status > >>> "rebalancing" when you

Re: Re: Servers Getting disconnected

2017-10-31 Thread Ted Yu
R's down to itself, and all clients work well. > I notice KAFKA-5153, it maybe same to mine. But it doesn't update for > a long time. > > > From: Ted Yu > Date: 2017-10-30 17:13 > To: users > Subject: Re: Servers Getting disconnected > Please take a look at KAFKA-4477

Re: Producer is blocking at second commitTransaction

2017-12-20 Thread Ted Yu
ProducerRecord<Long, String> record = new > ProducerRecord<>("test", 0, (long) 0, Long.toString(0)); > producer.send(record); > producer.commitTransaction(); > producer.beginTransaction(); > record = new ProducerRec

Re: Kafka 2.11-1.1.0 crashes brokers and brings cluster down on Windows

2018-05-14 Thread Ted Yu
rough debugger to find out which thread is >> locking the file. Apologise but it's a huge application so it might get me >> some time to get around it :) >> >> Please do update if you find something new. >> >> Regards, >> >> On 12 May 2018 at 22:15, Te

Re: Kafka 2.11-1.1.0 crashes brokers and brings cluster down on Windows

2018-05-14 Thread Ted Yu
5:29,439] INFO Stopping serving logs in dir C:\kafka1 > (kafka.log.LogManager) > [2018-05-14 16:35:29,517] INFO [ExpirationReaper-1-topic]: Starting > (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper) > [2018-05-14 16:35:29,517] ERROR Shutdown broker because all log

Re: Kafka offset problem when using Spark Streaming...

2018-05-09 Thread Ted Yu
Can you give us more information: release of Spark and Kafka you're using anything interesting from broker / client logs - feel free to use pastebin to pass snippet if needed Thanks On Wed, May 9, 2018 at 3:42 AM, Pena Quijada Alexander < a.penaquij...@reply.it> wrote: > Hi all, > > We're

Re: Re: kafka streams with TimeWindows ,incorrect result

2018-04-27 Thread Ted Yu
Noticed a typo in streams in subject. Corrected it in this reply. Original message From: 杰 杨 Date: 4/27/18 1:28 AM (GMT-08:00) To: 杰 杨 , users Subject: Re: Re: kafka steams with TimeWindows ,incorrect result and I

Re: source code location for KSQL

2018-04-27 Thread Ted Yu
https://github.com/confluentinc/ksql FYI On Fri, Apr 27, 2018 at 4:23 PM, Henry Cai wrote: > I think KSQL is also open sourced, where is the source code location for > KSQL in github? > > Thanks. >

Re: Kafka 2.11-1.1.0 crashes brokers and brings cluster down on Windows

2018-05-12 Thread Ted Yu
There are two outstanding issues: KAFKA-6059 and KAFKA-6200 which bear some resemblance. Can you try to find how the other process uses the file being deleted ? https://superuser.com/questions/117902/find-out-which-process-is-locking-a-file-or-folder-in-windows

Re: Help Needed: Leadership Issue upon Kafka Upgrade (ZooKeeper 3.4.9)

2018-05-11 Thread Ted Yu
Was there any previous connectivity issue to 1.1.1.143:3888 before the upgrade ? I assume you have verified that connectivity between broker and 1.1.1.143 is Okay. Which zookeeper release are you running ? Cheers On Fri, May 11, 2018 at 3:16 PM, Raghav

Re: Consulting ReadOnlyKeyValueStore from Processor can lead to deadlock

2018-05-10 Thread Ted Yu
bq. the docs and CachingKVS behavior could improve I would agree. Pointing out the usage of ReadWriteLock and mentioning the withCachingDisabled() method in doc would help other developers. On Thu, May 10, 2018 at 11:21 AM, Steven Schlansker < sschlans...@opentable.com> wrote: > > > On May 10,

Re: Kafka consumer slowly consume the data

2018-05-11 Thread Ted Yu
bq. load on the node is increasing tremendously Can you do some profiling to see where the bottleneck was ? You can pastebin some stack traces. Which Kafka release do you use ? Thanks On Fri, May 11, 2018 at 6:41 AM, Karthick Kumar wrote: > Hi, > > I'm using tomcat

Re: Kafka Scala debugging and diagnostics

2018-05-17 Thread Ted Yu
16 May 2018 at 23:04, M. Manna <manme...@gmail.com> wrote: > > > Must have missed that in Kafka-run-class.bat which sets suspend=n by > > default. Thanks for pointing that out, giving it a try. > > > > > > > > And didn > > > > On Wed, 16 May 2018

Re: Kafka Scala debugging and diagnostics

2018-05-16 Thread Ted Yu
Have you set the following two environment variables: export KAFKA_DEBUG=y; export DEBUG_SUSPEND_FLAG=y; The above would freeze the process until debugger is attached. Cheers On Wed, May 16, 2018 at 2:46 PM, M. Manna wrote: > Hello, > > Is there anyone who can provide a

Re: NPE in low level Kafka topology

2018-06-18 Thread Ted Yu
Can you show the related code from OneToManyGroupedProcessor ? Thanks On Mon, Jun 18, 2018 at 4:29 AM, Frank Lyaruu wrote: > Hi, I've upgraded our 0.11 based stream application to the trunk version, > and I get an intermittent NPE. It's is quite a big topology, and I haven't > succeeded in

Re: Details of segment deletion

2018-06-12 Thread Ted Yu
Minor clarification (since new segment appeared twice) : bq. before a new one is deleted. The 'new one' (in the last sentence) would become old when another segment is created. Cheers On Tue, Jun 12, 2018 at 6:42 PM, Gwen Shapira wrote: > See below: > > On Mon, Jun 11, 2018 at 3:36 AM,

Re: INVALID_FETCH_SESSION_EPOCH after upgrade to 1.1.0

2018-06-12 Thread Ted Yu
Before Errors.INVALID_FETCH_SESSION_EPOCH is returned, FetchSession.scala would log the reason for the response. There are 3 cases, 2 with info log and 1 with debug log. Here is one code snippet: if (session.epoch != reqMetadata.epoch()) { debug(s"Created a new error

Re: INVALID_FETCH_SESSION_EPOCH after upgrade to 1.1.0

2018-06-13 Thread Ted Yu
13, 2018 at 9:08 AM, Mark Anderson wrote: > Ted > > I don't see any other INFO log messages so I assume that means it is the > DEBUG case I'm seeing? > > I don't have DEBUG enabled at the moment. > > Thanks > > On Wed, 13 Jun 2018, 00:2

Re: [VOTE] 1.0.2 RC1

2018-06-30 Thread Ted Yu
+1 Ran test suite. On Fri, Jun 29, 2018 at 10:02 PM, Matthias J. Sax wrote: > Hello Kafka users, developers and client-developers, > > This is the second candidate for release of Apache Kafka 1.0.2. > > This is a bug fix release addressing 27 tickets: >

Re: Partitions reassignment is failing in Kafka 1.1.0

2018-07-01 Thread Ted Yu
See KAFKA-7124 Cheers On Sun, Jul 1, 2018 at 3:26 AM, Debraj Manna wrote: > Is there any bug filed for this ? > > On Sun, Jul 1, 2018 at 6:37 AM, Ted Yu wrote: > > > Code snippet from ReassignPartitionsCommand.scala : > > > >

Re: Cannot run the inbuild zookeeper server.

2018-07-01 Thread Ted Yu
'Address already in use' indicates that there was some (zookeeper) server running on the same machine. Can you double check ? Thanks On Sun, Jul 1, 2018 at 4:45 AM, Vineet Herenj wrote: > Hello, > > I am not able to run the inbuild zookeeper server. > Following error shows: > >

Re: [VOTE] 0.10.2.2 RC1

2018-06-29 Thread Ted Yu
+1 Ran test suite. Checked signatures. On Fri, Jun 29, 2018 at 10:21 AM, Jason Gustafson wrote: > +1 (binding). I checked release notes, documentation, and went through the > quickstart. > > Thanks Matthias! > > On Fri, Jun 22, 2018 at 6:43 PM, Matthias J. Sax > wrote: > > > Hello Kafka

Re: Partitions reassignment is failing in Kafka 1.1.0

2018-06-29 Thread Ted Yu
Seems to be related to KIP-113. server.properties didn't go thru. Do you mind pastebin'ing its content ? If you can pastebin logs from broker, that should help. Thanks On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna wrote: > Hi > > I altered a topic like below in kafka 1.1.0 > >

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

2018-06-27 Thread Ted Yu
hods I listed. > > For example: > > windowCounts > .suppress( > Suppression > .suppressLateEvents(Duration.ofMinutes(10)) > .suppressIntermediateEvents( > IntermediateSuppression.emitAfter(Duration.ofMinutes(10)) > )

Re: [VOTE] 1.1.1 RC1

2018-06-22 Thread Ted Yu
+1 Ran test suite. Checked signatures. On Fri, Jun 22, 2018 at 12:14 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > +1 (non-binding) > > Built from source and ran quickstart successfully on Ubuntu (with Java 8). > > Thanks Dong! > --Vahid > > > > From: Dong Lin > To:

Re: [VOTE] 0.11.0.3 RC0

2018-06-23 Thread Ted Yu
+1 Checked signatures. Ran unit test suite. On Fri, Jun 22, 2018 at 4:56 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > +1 (non-binding) > > Built from source and ran quickstart successfully on Ubuntu (with Java 8). > > Thanks Matthias! > --Vahid > > > > > From: "Matthias J.

Re: [VOTE] 1.1.1 RC0

2018-06-19 Thread Ted Yu
+1 Ran unit test suite which passed. Checked signatures. On Tue, Jun 19, 2018 at 4:47 PM, Dong Lin wrote: > Re-send to kafka-clie...@googlegroups.com > > On Tue, Jun 19, 2018 at 4:29 PM, Dong Lin wrote: > > > Hello Kafka users, developers and client-developers, > > > > This is the first

  1   2   >