MQTT Support (Mosquitto + Mosquitto Kafka Bridge + Kafka)

Hi All, I'm working on the IoT stack with Kafka. I am using Mosquitto + Mosquitto Kafka Bridge + Kafka stack as I need the protocol support for MQTT. I need to scale this stack too. Please advise on this in terms of scalability and if any better approach to handle this. Thanks

Re: Kafka over Satellite links

Sat links induce roughly 250ms of latency, you have to tweak TCP Windows so you can saturate the link as much as you can, but besides that Sat links are rather common. On Mar 4, 2016 7:26 AM, "Jan" wrote: > Thanks for the input. > The IoT application could have a http

Re: Writing a Producer from Scratch

Stephen, There is a mailing list for kafka client developers that you may find useful: https://groups.google.com/forum/#!forum/kafka-clients The d...@kafka.apache.org mailing list might also be a good resource: http://kafka.apache.org/contact.html Lastly, do you

Fwd: Kafka Security

Hi, I am exploring on the Security capabilities of Kafka 0.9.1 but unable to use it successfully. I have set below configuration in my server.properties *allow.everyone.if.no.acl.found=false* *super.users=User:root;User:kafka* I created an ACL using below command *./kafka-acls.sh

Re: Kafka over Satellite links

Thanks for the input.  The IoT application could have a http ingestion receiver to funnel data into Kafka. However, I am thinking about Kafka's ability to be able to injest traffic directly from IoT devices.   Does anyone have any recommendations while using Kafka Connect between producer &  

Re: Consumer deadlock

Can you post some logs from the consumer? That should tell us what it's busy doing while hanging. You may have to enable DEBUG level. -Jason On Thu, Mar 3, 2016 at 5:02 PM, Muthukumaran K wrote: > Hi Jason, > > I am using 0.9 broker. > > One more observation. I had

RE: Consumer deadlock

Hi Jason, I am using 0.9 broker. One more observation. I had written producer code with 0.9 - Even with Producer code, I had hanging issue where send method was hanging requesting metadata. Thread-dump below "main" prio=6 tid=0x02238000 nid=0x1390 in Object.wait()

Re: About bootstrap.servers

Hi Tian, Removing the client dependence on Zookeeper has been one of the main goals of the Kafka team for a while now. It simplifies client development since it's one less dependence and one less remote system they have to manage interaction with. It also makes a lot of sense with the security

Re: Consumer deadlock

Hi there, Just to clarify, is the broker still on 0.8? Unfortunately, the new consumer needs 0.9. That probably would explain the hanging. -Jason On Thu, Mar 3, 2016 at 2:28 AM, Muthukumaran K wrote: > Ewen, > > By new Consumer API, you mean KafkaConsumer ? I have

Larger Size Error Message

Got the following error message with Kafka 0.8.2.1: [2016-02-26 20:33:43,025] INFO Closing socket connection to /x due to invalid request: Request of length 1937006964 is not valid, it is larger than the maximum size of 104857600 bytes. (kafka.network.Processor) Didn't send a large message at

Re: [sdc-user] Re: Having trouble to connect StreamSets to Kafka with Kerberos authentication

Hi Harikiran, One comment: `advertised.host.name` is not used if `advertised.listeners` is set and similarly `host.name` is not used if `listeners` is set. In general, the use of those properties is now discouraged in favour of listeners. There is a PR to make the documentation clearer:

Kafka | Unable to publish data to broker - ClosedChannelException

Hi I am trying to run simple kafka producer consumer example on HDP but facing below exception. [2016-03-03 18:26:38,683] WARN Fetching topic metadata with correlation id 0 for topics [Set(page_visits)] from broker [BrokerEndPoint(0,sandbox.hortonworks.com,9092)] failed

Re: Having trouble to connect StreamSets to Kafka with Kerberos authentication

Hi Michal, Can you succesfully connect to the SASL port without StreamSet? For example using the console consumer as explain here? http://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption (the end-to-end example is all the way at the end of the blog) This can

Re: [sdc-user] Re: Having trouble to connect StreamSets to Kafka with Kerberos authentication

Hi Michal, Can you please add the *advertised.listeners* and *advertised.host.name * properties in your kafka server config file 'server.properties_krb5'? For example, I have the following configuration in my working setup listeners=SASL_PLAINTEXT://:9092

Re: Fetch Response V1 incorrectly documented

Just the correct documentation. To be sure it won't change with next minor release. On 3 March 2016 at 17:53, Grant Henke wrote: > It looks like you are right, the throttle time does come first. I have a > WIP implementation (https://github.com/apache/kafka/pull/970) that >

Mirrormaker only publishing to a single partition on destination cluster

Hey! I'm using kafka 0.9.0.1 and trying to replicate a cluster from one datacenter to another. mirror-maker properly connects to my source cluster and consumes messages, but for some reason is only publishing to a single partition for my topic in the destination cluster. So all of my partitions

Re: [sdc-user] Re: Having trouble to connect StreamSets to Kafka with Kerberos authentication

Hi Michal, Are you able to write and read from the kerberized Kafka setup using the Kafka Console Producer and Consumer? I am taking a look at your configuration files. Thanks Hari. On Thu, Mar 3, 2016 at 8:09 AM Jonathan Natkins wrote: > Hey Michal, > > I'm cc'ing the

Re: Writing a Producer from Scratch

Oups didn't see your response. Sorry 2016-03-03 17:23 GMT+01:00 Péricé Robin : > Hi, > > Maybe you should look at this : https://github.com/edenhill/librdkafka > > Regards, > > Robin > > 2016-03-03 11:47 GMT+01:00 Hopson, Stephen : > >> Hi,

Re: Writing a Producer from Scratch

Hi, Maybe you should look at this : https://github.com/edenhill/librdkafka Regards, Robin 2016-03-03 11:47 GMT+01:00 Hopson, Stephen : > Hi, > > Not sure if this is the right forum for this question, but if it not I’m > sure someone will direct me to the proper

RE: Writing a Producer from Scratch

Hi Ben, Yes I have looked at that code. It relies heavily on either Windows or Linux / Unix libraries and even the C libraries are not compatible with the C compiler on my system (my mainframe is 36 bit WORD based) so there are a number of issues with sizes that I am aware of and have to deal

Re: Writing a Producer from Scratch

Hi Stephen, Have you checked out https://github.com/edenhill/librdkafka ? It might be what you need (I don't do C, so it might not be right for you) Regards, Ben On Thu, Mar 3, 2016 at 10:47 AM, Hopson, Stephen < stephen.hop...@gb.unisys.com> wrote: > Hi, > > Not sure if this is the right

Re: Having trouble to connect StreamSets to Kafka with Kerberos authentication

Hey Michal, I'm cc'ing the StreamSets user list, which might be able to get you some better StreamSets-specific answers. Thanks! Natty On Thursday, March 3, 2016, Michał Kabocik wrote: > Dears, > > I’m Middleware Engineer and I’m trying to configure secure Kafka

Re: Fetch Response V1 incorrectly documented

It looks like you are right, the throttle time does come first. I have a WIP implementation (https://github.com/apache/kafka/pull/970) that generates the protocol docs based on the protocol specification in the code and the output for fetch response v2 is: > Fetch Response (Version: 2) =>

Writing a Producer from Scratch

Hi, Not sure if this is the right forum for this question, but if it not I'm sure someone will direct me to the proper one. Also, I am new to Kafka (but not new to computers). I want to write a kafka producer client for a Unisys OS 2200 mainframe. I need to write it in C, and since I have no

Re: migrating the main-page docs to gitbook format

I love that too +1 2016-03-02 21:15 GMT-05:00 Christian Posta : > For sure! Will take a look! > > On Wednesday, March 2, 2016, Gwen Shapira wrote: > > > Hey! > > > > Yes! We'd love that too! Maybe you want to help us out with > >

RE: Consumer deadlock

Ewen, By new Consumer API, you mean KafkaConsumer ? I have an issue with a poll in 0.9.0.1. poll hangs indefinitely even with the timeout Following is the consumer code which I am using. Any pointers would be helpful public class ConsumerLoop implements Runnable { private final

Re: Consumer deadlock

Yes, mapping an iterator interface to a networked service can have some drawbacks :) Providing a simple, familiar interface to Kafka is challenging -- if you didn't block and hasNext() returned false, what would you do after that? That return value doesn't indicate that the iterable for a Kafka

Re: Consumer deadlock

Thank you Ewan, I'll give it a shot, but I am still puzzled as to the reasoning behind implementing hasNext() in a manner that it can block. Isn't that counter to the whole point of the method? You either have it or you not. Blocking in the off chance one may have it in some time in the future

Re: Announcing rdkafka-dotnet - C# Apache Kafka client

Andreas, Thanks, this looks great and I think its leveraging the existing functionality available in librdkafka is a great choice. I've added this to the wiki page for clients: https://cwiki.apache.org/confluence/display/KAFKA/Clients -Ewen On Wed, Mar 2, 2016 at 12:07 PM, Andreas Heider

Re: Consumer deadlock

Take a look at the consumer.timeout.ms setting if you don't want the iterator to block indefinitely. And a better long term solution is to switch to the new consumer, but that obviously requires much more significant code changes. The new consumer API is a single-threaded poll-based API where you

Fetch Response V1 incorrectly documented

It seems that Fetch Response V1 is not correctly documented: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol It says the response should be: FetchResponse => [TopicName [Partition ErrorCode HighwaterMarkOffset MessageSetSize MessageSet]] ThrottleTime But it

Re: session.timeout.ms limit - Kafka Consumer

In fact, KIP-41 has been implemented in trunk -- see https://issues.apache.org/jira/browse/KAFKA-3007. Testing against a version including that change would be greatly appreciated to ensure it fully addresses the problems you're seeing. -Ewen On Wed, Mar 2, 2016 at 7:00 AM, Olson,Andrew