poll(0) after subscribe

[Mansi Shah]

Hi, 

I had a question around expected behavior / API expectations.

Is subscribe a lazy call? What I mean by that is if I subscribe to a topic and 
then call poll(0), then is this first poll guaranteed to never return data? 
Will the data pre-fetch not start as soon as subscribe is called? Is this 
behavior also true for assign calls?

I ask this because the 0.9.0 connect code does the following - 

subscribe(topicName); 
poll(0); <— expect this call to not return any data
and then it starts its actual data consuming poll iterator. 


In the above case we can have potential data loss if the that first poll(0) 
actually did return some records. I see this code has been restructured for 
0.10.0 timeframe.

Thanks in advance for any help on this subject !!
Mansi. 

java.nio.channels.ClosedChannelException

[Shivpriya Tamboskar]

Hello community,
 I am running a simple consumer for kafka such as this

   int timeout = 8;
   int bufferSize = 64*1024;
   consumer = new SimpleConsumer(host, port,timeout, bufferSize, clientName);

This runs fine for a couple for hours but I get an exception later on 
kafka.consumer.SimpleConsumer: Reconnect due to socket error: 
java.nio.channels.ClosedChannelException

and consumer stops ... has anyone faced this problem before ?

Thanks
Shivpriya

Re: Kafka broker decommission steps

[Alexis Midon]

The command line tool that ships with Kafka is error prone.

Our standard procedure is:
1. spin up the new broker
2. use `kafkat drain  [--brokers ]
3. shut down old broker

The `drain` command will generate and submit a partition assignment plan
where the new broker id replaces the old one. It's pretty much a "gsub(old,
new)".

We do it regularly. It's almost a mundane operation. The only challenge is
the volume of data being transferred over the network. Since there is no
throttling mechanism, the network is sometime saturated which might impact
other consumers/producers

See https://github.com/airbnb/kafkat





On Fri, Mar 4, 2016 at 7:28 AM Todd Palino  wrote:

> To answer your questions…
>
> 1 - Not in the way you want it to. There is a setting for automatic leader
> election (which I do not recommend anyone use at this time), but all that
> does is pick which of the currently assigned replicas should be the leader.
> It does not reassign partitions from one broker to another. Kafka does not
> have a facility for doing this automatically.
>
> 2 - No. The most you can do is move all the partitions off and then
> immediately shut down the broker process. Any broker that is live in the
> cluster can, and will, get partitions assigned to it by the controller.
>
> For what you want to do, you need you use the partition reassignment
> command line tool that ships with Kafka to reassign partitions from the old
> broker to the new one. Once that is complete, you can double check that the
> old broker has no partitions left and shut it down. I have a tool that we
> use internally to make this a lot easier, and I’m in the process of getting
> a repository set up to make it available via open source. It allows for
> more easily removing and adding brokers, and rebalancing partitions in a
> cluster without having to craft the reassignments by hand.
>
> -Todd
>
>
> On Fri, Mar 4, 2016 at 5:07 AM, Muqtafi Akhmad 
> wrote:
>
> > dear Kafka users,
> >
> > I have some questions regarding decommissioning kafka broker node and
> > replacing it with the new one. Lets say that we have three broker nodes
> and
> > each topic in Kafka has replication factor = 3, we upgrade one node with
> > the following steps :
> > 1. add one broker node to cluster
> > 2. shutdown old broker node
> >
> > My questions are
> > 1. When we add one new broker to the cluster will it trigger Kafka topic
> /
> > group leadership rebalance?
> > 2. Is there any way to disable the to-be-decommissioned node to hold no
> > topic/group leadership (acting as passive copy) so that it can be
> > decommissioned with minimal effect to Kafka clients?
> >
> > Thank you,
> >
> > --
> > Muqtafi Akhmad
> > Software Engineer
> > Traveloka
> >
>
>
>
> --
> *—-*
> *Todd Palino*
> Staff Site Reliability Engineer
> Data Infrastructure Streaming
>
>
>
> linkedin.com/in/toddpalino
>

Re: Kafka | Unable to publish data to broker - ClosedChannelException

[Banias H]

It sounds like an issue with the topic. You can try describe the page_vists
topic like:

./kafka-topics.sh --zookeeper localhost:2181 --describe

and make sure it has a leader. If not, I would suggest creating a new topic
and try if you can send messages to it.


On Fri, Mar 4, 2016 at 3:34 AM, Shashi Vishwakarma  wrote:

> Hi
>
> I changed command to
>
> ./kafka-console-producer.sh --broker-list  sandbox.hortonworks.com:6667
> --topic page_visits
>
> Exception changed but no success.
>
>
> [2016-03-04 09:26:12,742] WARN Error while fetching metadata
> [{TopicMetadata for topic page_visits ->
> No partition metadata for topic page_visits due to
> kafka.common.LeaderNotAvailableException}] for topic [page_visits]: class
> kafka.common.LeaderNotAvailableException
>  (kafka.producer.BrokerPartitionInfo)
> [2016-03-04 09:26:12,754] WARN Error while fetching metadata
> [{TopicMetadata for topic page_visits ->
> No partition metadata for topic page_visits due to
> kafka.common.LeaderNotAvailableException}] for topic [page_visits]: class
> kafka.common.LeaderNotAvailableException
>  (kafka.producer.BrokerPartitionInfo)
> [2016-03-04 09:26:12,755] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: page_visits
> (kafka.producer.async.DefaultEventHandler)
> [2016-03-04 09:26:12,865] WARN Error while fetching metadata
> [{TopicMetadata for topic page_visits ->
> No partition metadata for topic page_visits due to
> kafka.common.LeaderNotAvailableException}] for topic [page_visits]: class
> kafka.common.LeaderNotAvailableException
>  (kafka.producer.BrokerPartitionInfo)
> [2016-03-04 09:26:12,873] WARN Error while fetching metadata
> [{TopicMetadata for topic page_visits ->
> No partition metadata for topic page_visits due to
> kafka.common.LeaderNotAvailableException}] for topic [page_visits]: class
> kafka.common.LeaderNotAvailableException
>  (kafka.producer.BrokerPartitionInfo)
> [2016-03-04 09:26:12,873] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: page_visits
> (kafka.producer.async.DefaultEventHandler)
> [2016-03-04 09:26:12,979] WARN Error while fetching metadata
> [{TopicMetadata for topic page_visits ->
> No partition metadata for topic page_visits due to
> kafka.common.LeaderNotAvailableException}] for topic [page_visits]: class
> kafka.common.LeaderNotAvailableException
>  (kafka.producer.BrokerPartitionInfo)
> [2016-03-04 09:26:12,985] WARN Error while fetching metadata
> [{TopicMetadata for topic page_visits ->
> No partition metadata for topic page_visits due to
> kafka.common.LeaderNotAvailableException}] for topic [page_visits]: class
> kafka.common.LeaderNotAvailableException
>  (kafka.producer.BrokerPartitionInfo)
> [2016-03-04 09:26:12,985] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: page_visits
> (kafka.producer.async.DefaultEventHandler)
> [2016-03-04 09:26:13,095] WARN Error while fetching metadata
> [{TopicMetadata for topic page_visits ->
> No partition metadata for topic page_visits due to
> kafka.common.LeaderNotAvailableException}] for topic [page_visits]: class
> kafka.common.LeaderNotAvailableException
>  (kafka.producer.BrokerPartitionInfo)
> [2016-03-04 09:26:13,107] WARN Error while fetching metadata
> [{TopicMetadata for topic page_visits ->
> No partition metadata for topic page_visits due to
> kafka.common.LeaderNotAvailableException}] for topic [page_visits]: class
> kafka.common.LeaderNotAvailableException
>  (kafka.producer.BrokerPartitionInfo)
> [2016-03-04 09:26:13,107] ERROR Failed to collate messages by topic,
> partition due to: Failed to fetch topic metadata for topic: page_visits
> (kafka.producer.async.DefaultEventHandler)
> [2016-03-04 09:26:13,215] WARN Error while fetching metadata
> [{TopicMetadata for topic page_visits ->
> No partition metadata for topic page_visits due to
> kafka.common.LeaderNotAvailableException}] for topic [page_visits]: class
> kafka.common.LeaderNotAvailableException
>  (kafka.producer.BrokerPartitionInfo)
> [2016-03-04 09:26:13,217] ERROR Failed to send requests for topics
> page_visits with correlation ids in [0,8]
> (kafka.producer.async.DefaultEventHandler)
> [2016-03-04 09:26:13,223] ERROR Error in handling batch of 1 events
> (kafka.producer.async.ProducerSendThread)
> kafka.common.FailedToSendMessageException: Failed to send messages after 3
> tries.
> at
>
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:91)
> at
>
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105)
> at
>
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88)
> at
>
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68)
> at scala.collection.immutable.Stream.foreach(Stream.scala:547)
> at
>
> kafka.p

Re: Mirrormaker only publishing to a single partition on destination cluster

[Stephen Powis]

Took me a bit but worked out my issue.  Appears like all of the messages in
my topics were being hashed to a single partition using the
DefaultPartitioner.  Ended up rolling a simple round robin partitioner and
contributed that back via this ticket:
https://issues.apache.org/jira/browse/KAFKA-

Thanks
Stephen

On Thu, Mar 3, 2016 at 11:33 AM, Stephen Powis 
wrote:

> Hey!
>
> I'm using kafka 0.9.0.1 and trying to replicate a cluster from one
> datacenter to another.  mirror-maker properly connects to my source cluster
> and consumes messages, but for some reason is only publishing to a single
> partition for my topic in the destination cluster.  So all of my partitions
> for the topic are empty, except one, which contains everything from the
> source cluster.
>
> Has anyone seen this behavior before?  I must have something
> misconfigured, but am unable to figure it out from reviewing the online
> docs.
>
> Thanks!
> Stephen
>

Re: [sdc-user] Re: Having trouble to connect StreamSets to Kafka with Kerberos authentication

[Harikiran Nayak]

Hi Michal,

The configuration in consumer.properties is not correct.
The 'sasl.kerberos.service.name' option expects the kerberos principal that
Kafka runs as.
In your case it should be '*sasl.kerberos.service.name
=*kafka*' *

Can you please test using the Kafka Console Producer as well?
This will make sure that your kerberos setup is right.

Secondly, you must specify the same 'security.protocol' and '
sasl.kerberos.service.name' options in the Kafka Consumer stage in
StreamSets pipeline.
See attached snapshot that shows how to specify these properties.

Thanks
Hari.



*[image: Screen Shot 2016-03-04 at 10.32.42 AM.png]*

On Fri, Mar 4, 2016 at 7:16 AM Michał Kabocik 
wrote:

> Dear Hari,
>
> Thank you for your reply.
>
> Replying to your questions:
> Yes, I have all needed entries in etc/hosts and hosts can 'see' each other.
> I followed your suggestion and added mentioned entries in
> server.properties_krb5. Now when starting Kafka Broker I see:
> listeners = PLAINTEXT://:9092,SASL_PLAINTEXT://:9093
> advertised.listeners = PLAINTEXT://:9092,SASL_PLAINTEXT://:9093
> sasl.kerberos.service.name = kafka
> advertised.host.name = plx164h.xx.xxx.xx
>
> Unfortunately it didn't help. Error in StreamSets is the same. I've tried
> to use built-in kafka console consumer and also not succeded. Here is my
> config:
>
> On host A I have Kafka broker which is running with the config from
> previous email. On host B, I have another Kafka from which I used console
> consumer with following config:
>
> kafka_client_jaas.conf:
> KafkaClient {
> com.sun.security.auth.module.Krb5LoginModule required
> useKeyTab=true
> storeKey=true
> keyTab="/etc/security/keytabs/kafka_client.service.keytab"
> principal="client/10.xxx.xxx...@hdpcybersecacc.xx.xx";
> };
>
> consumer.properties:
> security.protocol=SASL_PLAINTEXT
> sasl.kerberos.service.name=client
>
> I'm starting console consumer with the command:
> ./bin/kafka-console-consumer.sh --bootstrap-server plx164h:9093 --topic
> streamsets2 --new-consumer --consumer.config consumer.properties
>
> When started, there is no error, console consumer seems to work fine, but
> when producing to this topic, no messages are read.
> From kerberos side everything looks correct:
>
> Mar 04 16:00:31 lxhnlxx.xxx.xx krb5kdc[16307](info): AS_REQ (4 etypes {18
> 17 16 23}) 10.xxx.xxx.72
> :
> ISSUE: authtime 1457103631, etypes {rep=18 tkt=18 ses=18},
> client/10.xxx....@hdpcybersecacc.xx.xx for
> krbtgt/hdpcybersecacc.xx...@hdpcybersecacc.xx.xx
> Mar 04 16:00:31 lxhnlxx.xxx.xx krb5kdc[16307](info): TGS_REQ (4 etypes {18
> 17 16 23}) 10.xxx.xxx.72
> :
> ISSUE: authtime 1457103631, etypes {rep=18 tkt=18 ses=18},
> client/10.xxx.xxx...@hdpcybersecacc.xx.xx for
> client/plx164h.xx...@hdpcybersecacc.xx.xx
>
> Could you please a look at this? Maybe you see configuration error?
>
> Kind regards,
> Michal
>
> W dniu czwartek, 3 marca 2016 17:49:03 UTC+1 użytkownik Harikiran Nayak
> napisał:
>
>> Hi Michal,
>>
>> Can you please add the *advertised.listeners* and *advertised.host.name
>> * properties in your kafka server config
>> file 'server.properties_krb5'?
>>
>> For example, I have the following configuration in my working setup
>>
>> listeners=SASL_PLAINTEXT://:9092
>> advertised.listeners=SASL_PLAINTEXT://:9092
>> host.name=kafka
>> advertised.host.name=kafka
>>
>> 'kafka' is the hostname on which the Kafka broker is running in my setup.
>> There is an entry for this host in '/etc/hosts' on the node where
>> StreamSets is running.
>>
>> Thanks
>> Hari.
>>
>> On Thu, Mar 3, 2016 at 8:19 AM Harikiran Nayak 
>> wrote:
>>
> Hi Michal,
>>>
>>> Are you able to write and read from the kerberized Kafka setup using the
>>> Kafka Console Producer and Consumer?
>>>
>>> I am taking a look at your configuration files.
>>>
>>> Thanks
>>> Hari.
>>>
>> On Thu, Mar 3, 2016 at 8:09 AM Jonathan Natkins 
>>> wrote:
>>>
>> Hey Michal,

 I'm cc'ing the StreamSets user list, which might be able to get you
 some better StreamSets-specific answers.

 Thanks!

>>> Natty

>>>

 On Thursday, March 3, 2016, Michał Kabocik 
 wrote:

> Dears,
>
> I’m Middleware Engineer and I’m trying to configure secure Kafka
> Cluster with SSL and Kerberos authentication with StreamSets, which will 
> be
> used for data injection to HDP.
>
> I have two Kafka Clusters; one with SSL enabled and there I
> successfully connected StreamSets to Kafka with SSL authentication, and
> second one with Kerberos authentication and here I’m facing with the
> problem:
>
> Both Kafka (with Zookeeper) and StreamSets are configured to
> auth

Uneven GC behavior between nodes

[Cees de Groot]

We're seeing something funny in one of our production clusters that I
cannot explain away. Everything works fine, but as we're ramping up on
Kafka, I really want to get at the root cause before we push a ton of
traffic through it :)

We have 6 nodes over three DCs in the cluster. Currently it's running a
light load of two topics, one with small (KB) messages, one with variable
sized (KB-MB) messages, both with 64 partitions and 3 replicas. All topics,
including __consumer_offsets, have been rebalanced with a script we wrote
to make sure that the replicas are spread out over the three datacenters
and that leadership is evenly balanced, so we can continue to operate if we
lose one DC. Producers use Consul to find an initial broker (round-robin
through the local DC), Consumers use the 0.9.0.1 client.

The funny thing is that in each DC, one broker graphs "normal" JVM heap
behavior - a sawtooth of the expected garbage creation/collection cycle.
The other one essentially stays flat. The flat-lining brokers also show
less incoming traffic when graphing the OS' received bytes. Everything else
- incoming, outgoing messages, etcetera, shows up as essentially the same
on the graphs.

I've been digging around for a bit, but can't find anything obvious that
would cause the differences in memory pressure. Assuming that Kafka brokers
pre-allocate buffers, I'd expect not too much garbage being generated. Is
the flatline the expected behavior and the sawtooth the unexpected one?
What could cause the difference?

Thanks for any pointers :-)


-- 

*Cees de Groot*
PRINCIPAL SOFTWARE ENGINEER
[image: PagerDuty logo] 
pagerduty.com
c...@pagerduty.com 
+1(416)435-4085

[image: Twitter] [image: FaceBook]
[image: Google+]
[image: LinkedIn]
[image: Blog]

Re: Kafka Applicability - Large Messages

[Cees de Groot]

1GB sounds like a tad steep, you may want to do some testing, as Kafka
needs to be told that such large messages can arrive and broker will then
pre-allocate buffers for that. Personally, I'd stop short of low megabytes,
anything bigger can be dropped off in e.g. S3 and then you just queue a
link for further processing.

I'm not saying it's impossible, Kafka handles large messages better than
most other tools out there, but you do want to do a test setup to make sure
that it'll handle the sort of traffic you fling at it in any case.

On Fri, Mar 4, 2016 at 4:26 AM, Mahesh Dharmasena 
wrote:

> We have a client with several thousand stores which send and receive
> messages to main system that resides on the headquarters.
>
> A single Store sends and receive around 50 to 100 messages per day.
>
> Average Message size could be from 2KB to 1GB.
>
> Please let me know whether I can adapt Apache Kafka for the solution?
>
>
> - Mahesh.
>



-- 

*Cees de Groot*
PRINCIPAL SOFTWARE ENGINEER
[image: PagerDuty logo] 
pagerduty.com
c...@pagerduty.com 
+1(416)435-4085

[image: Twitter] [image: FaceBook]
[image: Google+]
[image: LinkedIn]
[image: Blog]

Re: Kafka Applicability - Large Messages

[Vinoth Chandar]

I have used messages upto 20MB and while not ideal, works fairly well. But
if you are stepping into GB of data, you may need to chunk them up and
reassemble.


On Friday, March 4, 2016, Mahesh Dharmasena  wrote:

> We have a client with several thousand stores which send and receive
> messages to main system that resides on the headquarters.
>
> A single Store sends and receive around 50 to 100 messages per day.
>
> Average Message size could be from 2KB to 1GB.
>
> Please let me know whether I can adapt Apache Kafka for the solution?
>
>
> - Mahesh.
>

Kafka Applicability - Large Messages

[Mahesh Dharmasena]

We have a client with several thousand stores which send and receive
messages to main system that resides on the headquarters.

A single Store sends and receive around 50 to 100 messages per day.

Average Message size could be from 2KB to 1GB.

Please let me know whether I can adapt Apache Kafka for the solution?


- Mahesh.

Re: Kafka broker decommission steps

[Todd Palino]

To answer your questions…

1 - Not in the way you want it to. There is a setting for automatic leader
election (which I do not recommend anyone use at this time), but all that
does is pick which of the currently assigned replicas should be the leader.
It does not reassign partitions from one broker to another. Kafka does not
have a facility for doing this automatically.

2 - No. The most you can do is move all the partitions off and then
immediately shut down the broker process. Any broker that is live in the
cluster can, and will, get partitions assigned to it by the controller.

For what you want to do, you need you use the partition reassignment
command line tool that ships with Kafka to reassign partitions from the old
broker to the new one. Once that is complete, you can double check that the
old broker has no partitions left and shut it down. I have a tool that we
use internally to make this a lot easier, and I’m in the process of getting
a repository set up to make it available via open source. It allows for
more easily removing and adding brokers, and rebalancing partitions in a
cluster without having to craft the reassignments by hand.

-Todd


On Fri, Mar 4, 2016 at 5:07 AM, Muqtafi Akhmad 
wrote:

> dear Kafka users,
>
> I have some questions regarding decommissioning kafka broker node and
> replacing it with the new one. Lets say that we have three broker nodes and
> each topic in Kafka has replication factor = 3, we upgrade one node with
> the following steps :
> 1. add one broker node to cluster
> 2. shutdown old broker node
>
> My questions are
> 1. When we add one new broker to the cluster will it trigger Kafka topic /
> group leadership rebalance?
> 2. Is there any way to disable the to-be-decommissioned node to hold no
> topic/group leadership (acting as passive copy) so that it can be
> decommissioned with minimal effect to Kafka clients?
>
> Thank you,
>
> --
> Muqtafi Akhmad
> Software Engineer
> Traveloka
>



-- 
*—-*
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming



linkedin.com/in/toddpalino

Re: [sdc-user] Re: Having trouble to connect StreamSets to Kafka with Kerberos authentication

[Michał Kabocik]

Dear Hari,

Thank you for your reply.

Replying to your questions:
Yes, I have all needed entries in etc/hosts and hosts can 'see' each other.
I followed your suggestion and added mentioned entries in 
server.properties_krb5. Now when starting Kafka Broker I see:
listeners = PLAINTEXT://:9092,SASL_PLAINTEXT://:9093
advertised.listeners = PLAINTEXT://:9092,SASL_PLAINTEXT://:9093
sasl.kerberos.service.name = kafka
advertised.host.name = plx164h.xx.xxx.xx

Unfortunately it didn't help. Error in StreamSets is the same. I've tried 
to use built-in kafka console consumer and also not succeded. Here is my 
config:

On host A I have Kafka broker which is running with the config from 
previous email. On host B, I have another Kafka from which I used console 
consumer with following config:

kafka_client_jaas.conf:
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/etc/security/keytabs/kafka_client.service.keytab"
principal="client/10.xxx.xxx...@hdpcybersecacc.xx.xx";
};

consumer.properties:
security.protocol=SASL_PLAINTEXT
sasl.kerberos.service.name=client

I'm starting console consumer with the command:
./bin/kafka-console-consumer.sh --bootstrap-server plx164h:9093 --topic 
streamsets2 --new-consumer --consumer.config consumer.properties

When started, there is no error, console consumer seems to work fine, but 
when producing to this topic, no messages are read. 
>From kerberos side everything looks correct:

Mar 04 16:00:31 lxhnlxx.xxx.xx krb5kdc[16307](info): AS_REQ (4 etypes {18 
17 16 23}) 10.xxx.xxx.72 
:
 
ISSUE: authtime 1457103631, etypes {rep=18 tkt=18 ses=18}, 
client/10.xxx....@hdpcybersecacc.xx.xx for 
krbtgt/hdpcybersecacc.xx...@hdpcybersecacc.xx.xx
Mar 04 16:00:31 lxhnlxx.xxx.xx krb5kdc[16307](info): TGS_REQ (4 etypes {18 
17 16 23}) 10.xxx.xxx.72 
:
 
ISSUE: authtime 1457103631, etypes {rep=18 tkt=18 ses=18}, 
client/10.xxx.xxx...@hdpcybersecacc.xx.xx for 
client/plx164h.xx...@hdpcybersecacc.xx.xx

Could you please a look at this? Maybe you see configuration error?

Kind regards,
Michal

W dniu czwartek, 3 marca 2016 17:49:03 UTC+1 użytkownik Harikiran Nayak 
napisał:

> Hi Michal,
>
> Can you please add the *advertised.listeners* and *advertised.host.name 
> * properties in your kafka server config 
> file 'server.properties_krb5'?
>
> For example, I have the following configuration in my working setup
>
> listeners=SASL_PLAINTEXT://:9092
> advertised.listeners=SASL_PLAINTEXT://:9092
> host.name=kafka
> advertised.host.name=kafka
>
> 'kafka' is the hostname on which the Kafka broker is running in my setup. 
> There is an entry for this host in '/etc/hosts' on the node where 
> StreamSets is running.
>
> Thanks
> Hari.
>
> On Thu, Mar 3, 2016 at 8:19 AM Harikiran Nayak  > wrote:
>
>> Hi Michal,
>>
>> Are you able to write and read from the kerberized Kafka setup using the 
>> Kafka Console Producer and Consumer?
>>
>> I am taking a look at your configuration files.
>>
>> Thanks
>> Hari.
>>
>> On Thu, Mar 3, 2016 at 8:09 AM Jonathan Natkins > > wrote:
>>
>>> Hey Michal,
>>>
>>> I'm cc'ing the StreamSets user list, which might be able to get you 
>>> some better StreamSets-specific answers.
>>>
>>> Thanks!
>>> Natty
>>>
>>> On Thursday, March 3, 2016, Michał Kabocik >> > wrote:
>>>
 Dears,

 I’m Middleware Engineer and I’m trying to configure secure Kafka 
 Cluster with SSL and Kerberos authentication with StreamSets, which will 
 be 
 used for data injection to HDP.

 I have two Kafka Clusters; one with SSL enabled and there I 
 successfully connected StreamSets to Kafka with SSL authentication, and 
 second one with Kerberos authentication and here I’m facing with the 
 problem:

 Both Kafka (with Zookeeper) and StreamSets are configured to 
 authenticate via Kerberos. When starting all of them, I see in the logs, 
 that they are successfully authenticated (TGT granted etc.)

 I have two listeners defined in Kafka: 
 listeners=PLAINTEXT://:9092,SASL_PLAINTEXT://:9093. When starting Kafka, I 
 see Kafka listens on both, 9092 and 9093.

 When I connect StreamSets to Kafka on port 9092, everything works 
 smooth. But when I try to connect to port 9093, error occurs:

 KAFKA_41 - Could not get partition count for topic 'streamsets5' : 
 com.streamsets.pipeline.api.StageException: KAFKA_41 - Could not get 
 caseition count for topic 'streamsets5' : 
 org.apache.kafka.common.KafkaException: Failed to construct kafka consumer

 I see no errors in Kafka, in the log of StreamSets, there is only above 
 error visible. I attached major config files o

Re: [sdc-user] Re: Having trouble to connect StreamSets to Kafka with Kerberos authentication

[Michał Kabocik]

Dear Hari,

Thank you for your reply.

Replying to your questions:
Yes, I have all needed entries in etc/hosts and hosts can 'see' each other.
I followed your suggestion and added mentioned entries in 
server.properties_krb5. Now when starting Kafka Broker I see:
listeners = PLAINTEXT://:9092,SASL_PLAINTEXT://:9093
advertised.listeners = PLAINTEXT://:9092,SASL_PLAINTEXT://:9093
sasl.kerberos.service.name = kafka
advertised.host.name = plx164h.xx.xxx.xx

Unfortunately it didn't help. Error in StreamSets is the same. I've tried 
to use built-in kafka console consumer and also not succeded. Here is my 
config:

On host A I have Kafka broker which is running with the config from 
previous email. On host B, I have another Kafka from which I used console 
consumer with following config:

kafka_client_jaas.conf:
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/etc/security/keytabs/kafka_client.service.keytab"
principal="client/10.xxx.xxx...@hdpcybersecacc.xx.xx";
};

consumer.properties:
security.protocol=SASL_PLAINTEXT
sasl.kerberos.service.name=client

I'm starting console consumer with the command:
./bin/kafka-console-consumer.sh --bootstrap-server plx164h:9093 --topic 
streamsets2 --new-consumer --consumer.config consumer.properties

When started, there is no error, console consumer seems to work fine, but 
when producing to this topic, no messages are read. 
>From kerberos side everything looks correct:

Mar 04 16:00:31 lxhnl219.srv.pl.ing.net krb5kdc[16307](info): AS_REQ (4 
etypes {18 17 16 23}) 10.111.159.72: ISSUE: authtime 1457103631, etypes 
{rep=18 tkt=18 ses=18}, client/10.xxx....@hdpcybersecacc.xx.xx for 
krbtgt/hdpcybersecacc.xx...@hdpcybersecacc.xx.xx
Mar 04 16:00:31 lxhnl219.srv.pl.ing.net krb5kdc[16307](info): TGS_REQ (4 
etypes {18 17 16 23}) 10.111.159.72: ISSUE: authtime 1457103631, etypes 
{rep=18 tkt=18 ses=18}, client/10.xxx.xxx...@hdpcybersecacc.xx.xx for 
client/plx164h.xx...@hdpcybersecacc.xx.xx

Could you please a look at this? Maybe you see configuration error?

Kind regards,
Michal


W dniu czwartek, 3 marca 2016 17:49:03 UTC+1 użytkownik Harikiran Nayak 
napisał:

> Hi Michal,
>
> Can you please add the *advertised.listeners* and *advertised.host.name 
> * properties in your kafka server config 
> file 'server.properties_krb5'?
>
> For example, I have the following configuration in my working setup
>
> listeners=SASL_PLAINTEXT://:9092
> advertised.listeners=SASL_PLAINTEXT://:9092
> host.name=kafka
> advertised.host.name=kafka
>
> 'kafka' is the hostname on which the Kafka broker is running in my setup. 
> There is an entry for this host in '/etc/hosts' on the node where 
> StreamSets is running.
>
> Thanks
> Hari.
>
> On Thu, Mar 3, 2016 at 8:19 AM Harikiran Nayak  > wrote:
>
>> Hi Michal,
>>
>> Are you able to write and read from the kerberized Kafka setup using the 
>> Kafka Console Producer and Consumer?
>>
>> I am taking a look at your configuration files.
>>
>> Thanks
>> Hari.
>>
>> On Thu, Mar 3, 2016 at 8:09 AM Jonathan Natkins > > wrote:
>>
>>> Hey Michal,
>>>
>>> I'm cc'ing the StreamSets user list, which might be able to get you 
>>> some better StreamSets-specific answers.
>>>
>>> Thanks!
>>> Natty
>>>
>>> On Thursday, March 3, 2016, Michał Kabocik >> > wrote:
>>>
 Dears,

 I’m Middleware Engineer and I’m trying to configure secure Kafka 
 Cluster with SSL and Kerberos authentication with StreamSets, which will 
 be 
 used for data injection to HDP.

 I have two Kafka Clusters; one with SSL enabled and there I 
 successfully connected StreamSets to Kafka with SSL authentication, and 
 second one with Kerberos authentication and here I’m facing with the 
 problem:

 Both Kafka (with Zookeeper) and StreamSets are configured to 
 authenticate via Kerberos. When starting all of them, I see in the logs, 
 that they are successfully authenticated (TGT granted etc.)

 I have two listeners defined in Kafka: 
 listeners=PLAINTEXT://:9092,SASL_PLAINTEXT://:9093. When starting Kafka, I 
 see Kafka listens on both, 9092 and 9093.

 When I connect StreamSets to Kafka on port 9092, everything works 
 smooth. But when I try to connect to port 9093, error occurs:

 KAFKA_41 - Could not get partition count for topic 'streamsets5' : 
 com.streamsets.pipeline.api.StageException: KAFKA_41 - Could not get 
 caseition count for topic 'streamsets5' : 
 org.apache.kafka.common.KafkaException: Failed to construct kafka consumer

 I see no errors in Kafka, in the log of StreamSets, there is only above 
 error visible. I attached major config files of Kafka, Zookeeper and 
 StreamSets.

 Will greatly appreciate your help in solving this case!

 Kind regards,

>>>
>>>
>>> -- 
>>> Jonathan "Natty" Natkins
>>> StreamSets | Field

GetOffsetShell doesn't work with SASL enabled Kafka

[tao xiao]

Hi team,

I found that GetOffsetShell doesn't work with SASL enabled Kafka. I believe
this is due to old producer being used in GetOffsetShell. I want to if
there is any alternative to provide the same information with secure Kafka

Kafka version 0.9.0.1

Exception

% bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list
localhost:9092 --topic test --time -1
[2016-03-04 21:43:56,597] INFO Verifying properties
(kafka.utils.VerifiableProperties)
[2016-03-04 21:43:56,613] INFO Property client.id is overridden to
GetOffsetShell (kafka.utils.VerifiableProperties)
[2016-03-04 21:43:56,613] INFO Property metadata.broker.list is overridden
to localhost:9092 (kafka.utils.VerifiableProperties)
[2016-03-04 21:43:56,613] INFO Property request.timeout.ms is overridden to
1000 (kafka.utils.VerifiableProperties)
[2016-03-04 21:43:56,674] INFO Fetching metadata from broker
BrokerEndPoint(0,localhost,9092) with correlation id 0 for 1 topic(s)
Set(test) (kafka.client.ClientUtils$)
[2016-03-04 21:43:56,689] INFO Connected to localhost:9092 for producing
(kafka.producer.SyncProducer)
[2016-03-04 21:43:56,705] WARN Fetching topic metadata with correlation id
0 for topics [Set(test)] from broker [BrokerEndPoint(0,localhost,9092)]
failed (kafka.client.ClientUtils$)
java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:498)
at java.nio.HeapByteBuffer.getShort(HeapByteBuffer.java:304)
at kafka.api.ApiUtils$.readShortString(ApiUtils.scala:36)
at kafka.cluster.BrokerEndPoint$.readFrom(BrokerEndPoint.scala:52)
at
kafka.api.TopicMetadataResponse$$anonfun$1.apply(TopicMetadataResponse.scala:28)
at
kafka.api.TopicMetadataResponse$$anonfun$1.apply(TopicMetadataResponse.scala:28)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
at scala.collection.immutable.Range.foreach(Range.scala:166)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at kafka.api.TopicMetadataResponse$.readFrom(TopicMetadataResponse.scala:28)
at kafka.producer.SyncProducer.send(SyncProducer.scala:120)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
at kafka.tools.GetOffsetShell$.main(GetOffsetShell.scala:78)
at kafka.tools.GetOffsetShell.main(GetOffsetShell.scala)

Re: Kafka Security

[Ismael Juma]

Hi Martin,

I suggest reading
http://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption
for an end to end example of how to secure Kafka.

Ismael

On Fri, Mar 4, 2016 at 12:38 PM, Martin Gainty  wrote:

> Although authors suggest using existing Cloud security products such as
> Sentry (Cloudera) or Argus (Hortonworks) once Zookeeper adopted SASL
> integration ..kafka folk agreed SASL would be the best way to implement
> securing the following Kafka features :
>
> Authentication via SSL & Kerberos through SASLAuditingAuthorization
> through Unix-like users, permissions and ACLsEncryption over the wire
> (optional)It should be easy to enforce the use of security at a given site
> https://cwiki.apache.org/confluence/display/KAFKA/Security
> Unfortunately kafka-sasl authors suggested implementing SSO via PKCS7 is
> currently out-of-scope for pre 1.0 release
> Imagine working at a Global Bank where you need to sign on to 2+ different
> security realms to complete a transaction
> this may be too arduous for people in the real world who have been using
> one single-sign-on for years
> Unfortunately KAFKA-SASL-INTEGRATION project is still at 0.9 so current
> implementation is very beta (not at 1.0)
> https://cwiki.apache.org/confluence/display/KAFKA/Index
> CONCLUSION:If your client does not have Cloudera(Sentry) or
> Hortonworks(Argus) and desires the security features of
> SSLAuthentication/KerberosAuthentication, Auditing, Unix-Authorization,
> Wire-Encryption then KAFKA-SASL-Integration is the only suggested option
> anyone have a suggestion how to secure kafka?
>
> Martin
> __
>
>
>
> > Date: Fri, 4 Mar 2016 12:10:19 +0530
> > Subject: Fwd: Kafka Security
> > From: sudeepshekh...@gmail.com
> > To: users@kafka.apache.org
> >
> > Hi,
> >
> > I am exploring on the Security capabilities of Kafka 0.9.1 but unable to
> > use it successfully.
> >
> > I have set below configuration in my server.properties
> >
> > *allow.everyone.if.no.acl.found=false*
> > *super.users=User:root;User:kafka*
> >
> > I created an ACL using below command
> >
> > *./kafka-acls.sh --authorizer-properties zookeeper.connect=
> > --add --allow-principal User:imit --allow-host  --topic
> imit
> > --producer --consumer --group imit-consumer-group*
> >
> > and I see below response for it
> >
> > *Current ACLs for resource `Topic:imit`:*
> > *User:imit has Allow permission for operations: Describe from
> > hosts: *
> > *User:imit has Allow permission for operations: Read from hosts:
> > *
> > *User:imit has Allow permission for operations: Write from hosts:
> > *
> >
> > *Note:* Values mentioned in <> are replaced with some dummy values in the
> > question and used correctly while creating the ACL
> >
> > I have following observations:
> >
> > a) Though I define the rule for imit topic to access for a particular
> using
> > from a given host yet I can write to the topic from any host using any
> user
> > account.
> >
> > b) I am unable to read the messages from topic from any host or any user
> > account (even using the one for which I have defined the rules).
> >
> > I am running Kafka on RHEL 6.7 and all the users are local.
> >
> > Appreciate if someone can guide if I am missing any configuration
> > parameters or commands to manage authorization or if Kafka is behaving
> in a
> > weird way.
> >
> > Also where can I getting authorization related logs in Kafka?
> >
> >
> > Thanks & Regards,
> >
> > Sudeep
>
>

Kafka broker decommission steps

[Muqtafi Akhmad]

dear Kafka users,

I have some questions regarding decommissioning kafka broker node and
replacing it with the new one. Lets say that we have three broker nodes and
each topic in Kafka has replication factor = 3, we upgrade one node with
the following steps :
1. add one broker node to cluster
2. shutdown old broker node

My questions are
1. When we add one new broker to the cluster will it trigger Kafka topic /
group leadership rebalance?
2. Is there any way to disable the to-be-decommissioned node to hold no
topic/group leadership (acting as passive copy) so that it can be
decommissioned with minimal effect to Kafka clients?

Thank you,

-- 
Muqtafi Akhmad
Software Engineer
Traveloka

RE: Kafka Security

[Martin Gainty]

Although authors suggest using existing Cloud security products such as Sentry 
(Cloudera) or Argus (Hortonworks) once Zookeeper adopted SASL integration 
..kafka folk agreed SASL would be the best way to implement securing the 
following Kafka features :

Authentication via SSL & Kerberos through SASLAuditingAuthorization through 
Unix-like users, permissions and ACLsEncryption over the wire (optional)It 
should be easy to enforce the use of security at a given site
https://cwiki.apache.org/confluence/display/KAFKA/Security
Unfortunately kafka-sasl authors suggested implementing SSO via PKCS7 is 
currently out-of-scope for pre 1.0 release
Imagine working at a Global Bank where you need to sign on to 2+ different 
security realms to complete a transaction
this may be too arduous for people in the real world who have been using one 
single-sign-on for years
Unfortunately KAFKA-SASL-INTEGRATION project is still at 0.9 so current 
implementation is very beta (not at 
1.0)https://cwiki.apache.org/confluence/display/KAFKA/Index
CONCLUSION:If your client does not have Cloudera(Sentry) or Hortonworks(Argus) 
and desires the security features of SSLAuthentication/KerberosAuthentication, 
Auditing, Unix-Authorization, Wire-Encryption then KAFKA-SASL-Integration is 
the only suggested option 
anyone have a suggestion how to secure kafka?

Martin 
__ 

  


> Date: Fri, 4 Mar 2016 12:10:19 +0530
> Subject: Fwd: Kafka Security
> From: sudeepshekh...@gmail.com
> To: users@kafka.apache.org
> 
> Hi,
> 
> I am exploring on the Security capabilities of Kafka 0.9.1 but unable to
> use it successfully.
> 
> I have set below configuration in my server.properties
> 
> *allow.everyone.if.no.acl.found=false*
> *super.users=User:root;User:kafka*
> 
> I created an ACL using below command
> 
> *./kafka-acls.sh --authorizer-properties zookeeper.connect=
> --add --allow-principal User:imit --allow-host  --topic imit
> --producer --consumer --group imit-consumer-group*
> 
> and I see below response for it
> 
> *Current ACLs for resource `Topic:imit`:*
> *User:imit has Allow permission for operations: Describe from
> hosts: *
> *User:imit has Allow permission for operations: Read from hosts:
> *
> *User:imit has Allow permission for operations: Write from hosts:
> *
> 
> *Note:* Values mentioned in <> are replaced with some dummy values in the
> question and used correctly while creating the ACL
> 
> I have following observations:
> 
> a) Though I define the rule for imit topic to access for a particular using
> from a given host yet I can write to the topic from any host using any user
> account.
> 
> b) I am unable to read the messages from topic from any host or any user
> account (even using the one for which I have defined the rules).
> 
> I am running Kafka on RHEL 6.7 and all the users are local.
> 
> Appreciate if someone can guide if I am missing any configuration
> parameters or commands to manage authorization or if Kafka is behaving in a
> weird way.
> 
> Also where can I getting authorization related logs in Kafka?
> 
> 
> Thanks & Regards,
> 
> Sudeep
  

Kafka topic deletion still not working well

[Stevo Slavić]

Hell Apache Kafka community,

I'm still investigating an incident; from initial findings topic deletion
doesn't seem to work well still with Kafka 0.9.0.1, likely some edge case
not covered.

Before with 0.8.2.x it used to happen that non-lead replica would be stuck
in topic deletion process, and workaround was just to restart that node.

If I'm not mistaken, that edge case got (or at least is expected to be)
fixed in 0.9.0.1 via KAFKA-2937


Request to delete topic continued to be there in ZK even after whole
cluster restart - topic seemed not to exist, seemed to actually be deleted,
but request to delete topic would remain. Had to manually delete request
node in ZK.

When I have more details, and reproducible use case, will report back.

Kind regards,
Stevo Slavic.

Lost broker node

[michael.griffit...@baesystems.com]

Dear all,

On our development cluster we have a topic which only has a single replica. 
Since creating it we lost a node. By using the kafka-topics -describe command I 
can see that where the replica was on the lost node (broker #404), the leader 
has now become "-1".

Topic: alphaPartition: 0Leader: 399 Replicas: 399   
Isr: 399
Topic: alphaPartition: 1Leader: 400 Replicas: 400   
Isr: 400
Topic: alphaPartition: 2Leader: 402 Replicas: 402   
Isr: 402
Topic: alphaPartition: 3Leader: 403 Replicas: 403   
Isr: 403
Topic: alphaPartition: 4Leader: -1  Replicas: 404   
Isr: 404
Topic: alphaPartition: 5Leader: 405 Replicas: 405   
Isr: 405
Topic: alphaPartition: 6Leader: 406 Replicas: 406   
Isr: 406
Topic: alphaPartition: 7Leader: 398 Replicas: 398   
Isr: 398
Topic: alphaPartition: 8Leader: 399 Replicas: 399   
Isr: 399
Topic: alphaPartition: 9Leader: 400 Replicas: 400   
Isr: 400
Topic: alphaPartition: 10   Leader: 402 Replicas: 402   
Isr: 402
Topic: alphaPartition: 11   Leader: 403 Replicas: 403   
Isr: 403
Topic: alphaPartition: 12   Leader: -1  Replicas: 404   
Isr: 404
Topic: alphaPartition: 13   Leader: 405 Replicas: 405   
Isr: 405
Topic: alphaPartition: 14   Leader: 406 Replicas: 406   
Isr: 406
Topic: alphaPartition: 15   Leader: 398 Replicas: 398   
Isr: 398

I ran the ReassignPartitionsCommand, and set the broker list to 
"403,405,406,400,399,402,398" to reassign them away from 404 but the 
reassignments have been in-progress for quite a while now. Does broker 404 need 
to be up and alive to allow the reassignment to complete? Or can the 
reassignment occur without 404 being up and well and I'm therefore being 
impatient?

The updated kafka-topics output is now:

Topic: alphaPartition: 0Leader: 398 Replicas: 398   
Isr: 398
Topic: alphaPartition: 1Leader: 400 Replicas: 403,400   
Isr: 400
Topic: alphaPartition: 2Leader: 405 Replicas: 405   
Isr: 405
Topic: alphaPartition: 3Leader: 406 Replicas: 406   
Isr: 406
Topic: alphaPartition: 4Leader: -1  Replicas: 400,404   
Isr: 404
Topic: alphaPartition: 5Leader: 399 Replicas: 399   
Isr: 399
Topic: alphaPartition: 6Leader: 402 Replicas: 402   
Isr: 402
Topic: alphaPartition: 7Leader: 398 Replicas: 398   
Isr: 398
Topic: alphaPartition: 8Leader: 403 Replicas: 403   
Isr: 403
Topic: alphaPartition: 9Leader: 400 Replicas: 405,400   
Isr: 400
Topic: alphaPartition: 10   Leader: 406 Replicas: 406   
Isr: 406
Topic: alphaPartition: 11   Leader: 403 Replicas: 400,403   
Isr: 403
Topic: alphaPartition: 12   Leader: -1  Replicas: 399,404   
Isr: 404
Topic: alphaPartition: 13   Leader: 402 Replicas: 402   
Isr: 402
Topic: alphaPartition: 14   Leader: 398 Replicas: 398   
Isr: 398
Topic: alphaPartition: 15   Leader: 403 Replicas: 403   
Isr: 403

Many thanks,

Michael


Michael Griffiths
System Developer
BAE Systems Applied Intelligence
___

|  E: michael.griffit...@baesystems.com

BAE Systems Applied Intelligence, Surrey Research Park, Guildford, Surrey, GU2 
7RQ.
www.baesystems.com/ai

Please consider the environment before printing this email. This message should 
be regarded as confidential. If you have received this email in error please 
notify the sender and destroy it immediately. Statements of intent shall only 
become binding when confirmed in hard copy by an authorised signatory. The 
contents of this email may relate to dealings with other companies under the 
control of BAE Systems Applied Intelligence Limited, details of which can be 
found at http://www.baesystems.com/Businesses/index.htm.

Re: Kafka | Unable to publish data to broker - ClosedChannelException

[Shashi Vishwakarma]

Hi

I changed command to

./kafka-console-producer.sh --broker-list  sandbox.hortonworks.com:6667
--topic page_visits

Exception changed but no success.


[2016-03-04 09:26:12,742] WARN Error while fetching metadata
[{TopicMetadata for topic page_visits ->
No partition metadata for topic page_visits due to
kafka.common.LeaderNotAvailableException}] for topic [page_visits]: class
kafka.common.LeaderNotAvailableException
 (kafka.producer.BrokerPartitionInfo)
[2016-03-04 09:26:12,754] WARN Error while fetching metadata
[{TopicMetadata for topic page_visits ->
No partition metadata for topic page_visits due to
kafka.common.LeaderNotAvailableException}] for topic [page_visits]: class
kafka.common.LeaderNotAvailableException
 (kafka.producer.BrokerPartitionInfo)
[2016-03-04 09:26:12,755] ERROR Failed to collate messages by topic,
partition due to: Failed to fetch topic metadata for topic: page_visits
(kafka.producer.async.DefaultEventHandler)
[2016-03-04 09:26:12,865] WARN Error while fetching metadata
[{TopicMetadata for topic page_visits ->
No partition metadata for topic page_visits due to
kafka.common.LeaderNotAvailableException}] for topic [page_visits]: class
kafka.common.LeaderNotAvailableException
 (kafka.producer.BrokerPartitionInfo)
[2016-03-04 09:26:12,873] WARN Error while fetching metadata
[{TopicMetadata for topic page_visits ->
No partition metadata for topic page_visits due to
kafka.common.LeaderNotAvailableException}] for topic [page_visits]: class
kafka.common.LeaderNotAvailableException
 (kafka.producer.BrokerPartitionInfo)
[2016-03-04 09:26:12,873] ERROR Failed to collate messages by topic,
partition due to: Failed to fetch topic metadata for topic: page_visits
(kafka.producer.async.DefaultEventHandler)
[2016-03-04 09:26:12,979] WARN Error while fetching metadata
[{TopicMetadata for topic page_visits ->
No partition metadata for topic page_visits due to
kafka.common.LeaderNotAvailableException}] for topic [page_visits]: class
kafka.common.LeaderNotAvailableException
 (kafka.producer.BrokerPartitionInfo)
[2016-03-04 09:26:12,985] WARN Error while fetching metadata
[{TopicMetadata for topic page_visits ->
No partition metadata for topic page_visits due to
kafka.common.LeaderNotAvailableException}] for topic [page_visits]: class
kafka.common.LeaderNotAvailableException
 (kafka.producer.BrokerPartitionInfo)
[2016-03-04 09:26:12,985] ERROR Failed to collate messages by topic,
partition due to: Failed to fetch topic metadata for topic: page_visits
(kafka.producer.async.DefaultEventHandler)
[2016-03-04 09:26:13,095] WARN Error while fetching metadata
[{TopicMetadata for topic page_visits ->
No partition metadata for topic page_visits due to
kafka.common.LeaderNotAvailableException}] for topic [page_visits]: class
kafka.common.LeaderNotAvailableException
 (kafka.producer.BrokerPartitionInfo)
[2016-03-04 09:26:13,107] WARN Error while fetching metadata
[{TopicMetadata for topic page_visits ->
No partition metadata for topic page_visits due to
kafka.common.LeaderNotAvailableException}] for topic [page_visits]: class
kafka.common.LeaderNotAvailableException
 (kafka.producer.BrokerPartitionInfo)
[2016-03-04 09:26:13,107] ERROR Failed to collate messages by topic,
partition due to: Failed to fetch topic metadata for topic: page_visits
(kafka.producer.async.DefaultEventHandler)
[2016-03-04 09:26:13,215] WARN Error while fetching metadata
[{TopicMetadata for topic page_visits ->
No partition metadata for topic page_visits due to
kafka.common.LeaderNotAvailableException}] for topic [page_visits]: class
kafka.common.LeaderNotAvailableException
 (kafka.producer.BrokerPartitionInfo)
[2016-03-04 09:26:13,217] ERROR Failed to send requests for topics
page_visits with correlation ids in [0,8]
(kafka.producer.async.DefaultEventHandler)
[2016-03-04 09:26:13,223] ERROR Error in handling batch of 1 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:91)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68)
at scala.collection.immutable.Stream.foreach(Stream.scala:547)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67)
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:45)



Can you point where i am going wrong?

Thanks
Shashi

On Fri, Mar 4, 2016 at 2:48 AM, Banias H  wrote:

> Try changing the port like below.
>
> ./kafka-console-producer.sh --broker-list sandbox.hortonworks.com:
> *6667* --topic page_visits
>
> -B
>
> On Thu, Mar 3, 2016 at 12:45 PM, Shashi Vishwakarma <
> shashi.vi

RE: Writing a Producer from Scratch

[Hopson, Stephen]

Thanks James.
I will repost my question on dev mailing list and checkout the forum you 
mention. HTTP is a possibility but not the preferred option.

Cheers.
Steve

Stephen Hopson | Infrastructure Architect | Enterprise Solutions
Unisys | +44 (0)1908 805010 | +44 (0)7557 303321 | stephen.hop...@gb.unisys.com 



THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is for use only by the intended recipient. If you received this in 
error, please contact the sender and delete the e-mail and its attachments from 
all devices.

-Original Message-
From: James Cheng [mailto:jch...@tivo.com] 
Sent: 04 March 2016 06:55
To: users@kafka.apache.org
Subject: Re: Writing a Producer from Scratch

Stephen,

There is a mailing list for kafka client developers that you may find useful: 
https://groups.google.com/forum/#!forum/kafka-clients

The d...@kafka.apache.org mailing list might also 
be a good resource: http://kafka.apache.org/contact.html

Lastly, do you have any way to do HTTP calls on your platform? There exist some 
REST servers that you speak HTTP to and then they will produce to Kafka on your 
behalf. Here is one: http://docs.confluent.io/2.0.1/kafka-rest/docs/index.html

-James

On Mar 3, 2016, at 2:47 AM, Hopson, Stephen 
mailto:stephen.hop...@gb.unisys.com>> wrote:

Hi,
Not sure if this is the right forum for this question, but if it not I’m sure 
someone will direct me to the proper one.
Also, I am new to Kafka (but not new to computers).

I want to write a kafka producer client for a Unisys OS 2200 mainframe. I need 
to write it in C, and since I have no access to Windows / Unix / Linux 
libraries, I have to develop the interface at the lowest level.

So far, I have downloaded a kafka server with associated zookeeper (kafka 
_2.10-0.8.2.2). Note I have downloaded the Windows version and have it running 
on my laptop, successfully tested on the same laptop with the provided provider 
and consumer clients.

I have developed code to open a TCP session to the kafka server which appears 
to work and I have attempted to send a metadata request which does not appear 
to work. When I say it does not appear to work, I mean that I send the message 
and then I sit on a retrieve, which eventually times out ( I do seem to get one 
character in the receive buffer of 0235 octal). The message format I am using 
is the one described by the excellent document by Jay Creps / Gwen Shapira 
athttps://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
  However, it is not clear what level of kafka these message formats are 
applicable for.

Can anybody offer me any advice or suggestions as to how to progress?

PS is the CRC mandatory in the Producer messages?
Many thanks in advance.

Stephen Hopson | Infrastructure Architect | Enterprise Solutions
Unisys | +44 (0)1908 805010 | +44 (0)7557 303321 | 
stephen.hop...@gb.unisys.com



THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is for use only by the intended recipient. If you received this in 
error, please contact the sender and delete the e-mail and its attachments from 
all devices.




This email and any attachments may contain confidential and privileged material 
for the sole use of the intended recipient. Any review, copying, or 
distribution of this email (or any attachments) by others is prohibited. If you 
are not the intended recipient, please contact the sender immediately and 
permanently delete this email and any attachments. No employee or agent of TiVo 
Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by 
email. Binding agreements with TiVo Inc. may only be made by a signed written 
agreement.