Re: Does offsetsForTimes use createtime of logsegment file?

On Wed, Jan 4, 2017 at 11:54 PM, Vignesh wrote: > Hi, > > offsetsForTimes > KafkaConsumer.html#offsetsForTimes(java.util.Map)> > function > returns offset for a given timestamp. Does it use

Re: MirrorMaker - Topics Identification and Replication

s wrong in my understanding. > > Thanks > > On Tue, 3 Jan 2017 at 23:24 Ewen Cheslack-Postava <e...@confluent.io> > wrote: > > > Yes, the consumer will pick up the new topics when it refreshes metadata > > (defaults to every 5 min) and start subscrib

Re: One big kafka connect cluster or many small ones?

ems reasonable for most folks since *ideally* you are *somewhat* standardized on a common serialization format). -Ewen > > On 6 January 2017 at 1:54:10 pm, Ewen Cheslack-Postava (e...@confluent.io) > wrote: > > On Thu, Jan 5, 2017 at 3:12 PM, Stephane Maarek < > steph...@simpl

Re: Apache Kafka integration using Apache Camel

More generally, do you have any log errors/messages or additional info? It's tough to debug issues like this from 3rd party libraries if they don't provide logs/exception info that indicates why processing a specific message failed. -Ewen On Thu, Jan 5, 2017 at 8:29 PM, UMESH CHAUDHARY

Re: Kafka Connect offset.storage.topic not receiving messages (i.e. how to access Kafka Connect offset metadata?)

On Thu, Jan 5, 2017 at 11:30 AM, Phillip Mann wrote: > I am working on setting up a Kafka Connect Distributed Mode application > which will be a Kafka to S3 pipeline. I am using Kafka 0.10.1.0-1 and Kafka > Connect 3.1.1-1. So far things are going smoothly but one aspect that

Re: Json to JDBC using Kafka JDBC connector Sink

Anything with a table structure is probably not going to handle schemaless data (i.e. JSON) very well without some extra help -- tables usually expect schemas and JSON doesn't have a schema. As it stands today, the JDBC sink connector will probably not handle your use case. To send schemaless

Re: Producer connect timeouts

ult is > certainly very surprising behavior. It would also be nice not to have to > coordinate request timeouts, retries, and the max block configuration with > system-level configs. > > > On Sat, Dec 17, 2016 at 6:55 PM, Ewen Cheslack-Postava <e...@confluent.io> > wrote

Re: TLS

Ruben, There are step-by-step instructions explained here: http://docs.confluent. io/3.1.1/kafka/security.html For the purposes of configuring Kafka, the JAAS details basically boil down to a security configuration in a security configuration file. -Ewen On Mon, Dec 19, 2016 at 8:40 AM, Ruben

Re: Is it possible for consumers within a single consumer group to have different subscriptions?

It is possible for them to have different subscriptions. Consumers will only be assigned partitions from topics to which they are subscribed. So if you need to modify your app use data from an additional topic, you can safely do a rolling deploy of the updated version and during the period where

Re: Are there Connector artifacts in Confluent or any other Maven repository?

Yes, these get published to Confluent's maven repository. Follow the instructions here http://docs.confluent.io/current/installation.html#installation-maven for adding the Confluent maven repository to your project and then add a dependency for the connector to your project (e.g. for that

Re: kafka connector for mongodb as a source

There is some log noise in there from Reflections, but it does look like your connector & task are being created: [2017-03-27 18:33:00,057] INFO Instantiated task mongodb-0 with version 0.10.0.1 of type org.apache.kafka.connect.mongodb.MongodbSourceTask

Re: Kafka Connect behaves weird in case of zombie Kafka brokers. Also, zombie brokers?

Is that node the only bootstrap broker provided? If the Connect worker was pinned to *only* that broker, it wouldn't have any chance of recovering correct cluster information from the healthy brokers. It sounds like there was a separate problem as well (the broker should have figured out it was

Re: offset.storage.filename configuration in kafka-connect-hdfs

The offset file is marked as required so we can fail early & fast, but if you only run sink connectors then offsets will be stored in Kafka's normal offsets topic and the file will never need to be created. (And the HDFS connector is even more unusual in that it doesn't even rely on Kafka offsets

Re: offset commitment from another client

Consumers are responsible for committing offsets, not brokers. See http://kafka.apache.org/documentation.html#design_consumerposition for more of an explanation of how this is tracked. The brokers help coordinate this/store the offsets, but it is the consumers that decide when to commit offsets

Re: even if i pass key no change in partition

Do you have more than 1 partition? You may have an auto-created topic with only 1 partition, in which case the partition of messages will *always* be the same, regardless of key. -Ewen On Fri, Mar 24, 2017 at 5:52 AM, Laxmi Narayan wrote: > Hi, > > I am passing key in

Re: [DISCUSS] KIP-174 - Deprecate and remove internal converter configs in WorkerConfig

now if you have some additional thoughts on this. > > Regards, > Umesh > > > > On Wed, 26 Jul 2017 at 09:27 Ewen Cheslack-Postava <e...@confluent.io> > wrote: > >> Umesh, >> >> Thanks for the KIP. Straightforward and I think it's a good change.

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

Vahid, Thanks for the KIP. I think we're mostly in violent agreement that the lack of any Write permissions on consumer groups is confusing. Unfortunately it's a pretty annoying issue to fix since it would require an increase in permissions. More generally, I think it's unfortunate because by

Re: Kafka Connect Embedded API

The vast majority of KIP-26 has been implemented. Unfortunately, the embedded API is still one of the gaps that has not yet been implemented. It likely requires some additional design work as only a prototype API was proposed in the KIP describing the framework as a whole. -Ewen On Wed, Jul 12,

Re: Kafka Connect distributed mode rebalance

Btw, if you can share, I would be curious what connectors you're using and why you need so many. I'd be interested if a modification to the connector could also simplify things for you. -Ewen On Wed, Jul 26, 2017 at 12:33 AM, Ewen Cheslack-Postava <e...@confluent.io> wrote: > Stephen,

Re: Kafka Connect distributed mode rebalance

Stephen, Cool, that is a *lot* of connectors! Regarding rebalances, the reason this happens is that Kafka Connect is trying to keep the total work of the cluster balanced across the workers. If you add/remove connectors or the # of workers change, then we need to go through another round

Re: Kafka connector throughput reduction upon avro schema change

What is your setting for schema.compatibility? I suspect the issue is probably that it is defaulting to NONE which would cause the connector to roll a new file when the schema changes (which will be frequent with data that is interleaved with different schemas). If you set it to BACKWARDS then

Re: struggling with runtime Schema in connect

Stephen's explanation is great and accurate :) One of the design goals for Kafka Connect was to not rely on any specific serialization format since that is really orthogonal to getting/sending data from/to other systems. We define the generic *runtime* data API, which is what you'll find in the

Re: [DISCUSS] KIP-174 - Deprecate and remove internal converter configs in WorkerConfig

Umesh, Thanks for the KIP. Straightforward and I think it's a good change. Unfortunately it is hard to tell how many people it would affect since we can't tell how many people have adjusted that config, but I think this is the right thing to do long term. A couple of quick things that might be

Re: [DISCUSS] KIP-174 - Deprecate and remove internal converter configs in WorkerConfig

details to > it. > > Regards, > Umesh > > On Mon, 31 Jul 2017 at 21:51 Ewen Cheslack-Postava <e...@confluent.io> > wrote: > >> On Sun, Jul 30, 2017 at 10:21 PM, UMESH CHAUDHARY <umesh9...@gmail.com> >> wrote: >> >>> Hi Ewen, >>> Than

Re: struggling with runtime Schema in connect

t than any other serialization library/format! -Ewen On Wed, Jul 26, 2017 at 6:11 AM, Koert Kuipers <ko...@tresata.com> wrote: > just out of curiosity, why does kafka streams not use this runtime data api > defined in kafka connect? > > On Wed, Jul 26, 2017 at 3:10 AM,

Re: Reg: [VOTE] KIP 157 - Add consumer config options to streams reset tool

+1 (binding) I mentioned this in the PR that triggered this: > KIP is accurate, though this is one of those things that we should probably get a KIP for a standard set of config options across all tools so additions like this can just fall under the umbrella of that KIP... I think it would be

Re: Kafka Connect CPU spikes bring down Kafka Connect workers

On Mon, May 15, 2017 at 2:06 PM, Phillip Mann wrote: > Currently, Kafka Connect experiences a spike in CPU usage which causes > Kafka Connect to crash. What kind of crash? Can you provide an error or stacktrace? > There is really no useful information from the logs to

Re: Kafka broker startup issue

Version 2 of UpdateMetadataRequest does not exist in version 0.9.0.1. This suggests that you have a broker with a newer version of Kafka running against the same ZK broker. Do you have any other versions running? Or is it possible this is a shared ZK cluster and you're not using a namespace within

Re: [DISCUSS] KIP-174 - Deprecate and remove internal converter configs in WorkerConfig

wrote: > Thanks Ewen, > I just edited the KIP to reflect the changes. > > Regards, > Umesh > > On Wed, 9 Aug 2017 at 11:00 Ewen Cheslack-Postava <e...@confluent.io> > wrote: > >> Great, looking good. I'd probably be a bit more concrete about the >> Propose

[VOTE] 1.0.1 RC0

Hello Kafka users, developers and client-developers, Sorry for a bit of delay, but I've now prepared the first candidate for release of Apache Kafka 1.0.1. This is a bugfix release for the 1.0 branch that was first released with 1.0.0 about 3 months ago. We've fixed 46 significant issues since

[VOTE] 1.0.1 RC1

/documentation.html * Protocol: http://kafka.apache.org/10/protocol.html Thanks, Ewen Cheslack-Postava

Re: [VOTE] 1.0.1 RC1

And of course I'm +1 since I've already done normal release validation before posting this. -Ewen On Mon, Feb 12, 2018 at 10:15 AM, Ewen Cheslack-Postava <e...@confluent.io> wrote: > Hello Kafka users, developers and client-developers, > > This is the second candidate for re

Re: [VOTE] 1.0.1 RC1

sn't been updated yet: > > https://repository.apache.org/content/groups/staging/org/ > apache/kafka/kafka-clients/ > > On Mon, Feb 12, 2018 at 10:16 AM, Ewen Cheslack-Postava <e...@confluent.io > > > wrote: > > > And of course I'm +1 since I've already done normal

Re: [VOTE] 1.0.1 RC0

om source and running the quickstart were successful on Ubuntu > and Windows 10. > > Thanks for running the release. > --Vahid > > > > From: Ewen Cheslack-Postava <e...@confluent.io> > To: d...@kafka.apache.org, users@kafka.apache.org, > kafka-clie...

Re: [VOTE] 1.0.1 RC1

t (org.apache.kafka:streams-quickstart-java:1.0.1) > > Something i'm missing? > > Thanks, > Damian > > On Tue, 13 Feb 2018 at 10:16 Manikumar <manikumar.re...@gmail.com> wrote: > > > +1 (non-binding) > > > > ran quick-start, unit tests on the src. &

[VOTE] 1.0.1 RC2

/kafka/tree/1.0.1-rc2 * Documentation: http://kafka.apache.org/10/documentation.html * Protocol: http://kafka.apache.org/10/protocol.html /** Thanks, Ewen Cheslack-Postava

Re: [VOTE] 1.0.1 RC1

t; >> > > Satish. > >> > > > >> > > > >> > > On Tue, Feb 13, 2018 at 11:30 PM, Damian Guy <damian@gmail.com> > >> > wrote: > >> > > > >> > > > +1 > >> > > > > >&

[ANNOUNCE] Apache Kafka 1.0.1 Released

, Ewen Cheslack-Postava, Filipe Agapito, fredfp, Guozhang Wang, huxihx, Ismael Juma, Jason Gustafson, Jeremy Custenborder, Jiangjie (Becket) Qin, Joel Hamill, Konstantine Karantasis, lisa2lisa, Logan Buckley, Manjula K, Matthias J. Sax, Nick Chiu, parafiend, Rajini Sivaram, Randall Hauch, Robert

Re: Kafka Connect task re-balance repeatedly

The log is showing that the Connect worker is trying to make sure it has read the entire log and gets to offset 119, but some other worker says it has read to offset 169. The two are in inconsistent states, so the one that seems to be behind will not start work with potentially outdated

Re: [VOTE] 2.0.1 RC0

+1 -Ewen On Thu, Nov 1, 2018 at 10:10 AM Manikumar wrote: > We were waiting for the system test results. There were few failures: > KAFKA-7579, KAFKA-7559, KAFKA-7561 > they are not blockers for 2.0.1 release. We need more votes from > PMC/committers :) > > Thanks Stanislav! for the system

<    1   2   3   4