Re: Subcribe kafka

[Matthias J. Sax]

Follow instructions here: https://kafka.apache.org/contact

On 9/21/18 5:01 AM, ?? wrote:
> Hi:
>   I am new a user??I want to subscribe Kafka
> 



signature.asc
Description: OpenPGP digital signature

Subcribe kafka

[??????]

Hi:
  I am new a user??I want to subscribe Kafka

Re: Problems trying to make kafka 'rack-aware'

[Bryan Duggan]

Hi Eno,

many thanks for trying that. That is very helpful for me.

That basic check didn't work for me but I have since discovered what my 
issue was. Despite using a version of kafka that supports rack-awareness 
we have been deliberately setting 'inter.broker.protocol.version' to an 
older version (due to various issues with some of our consumers). When I 
update this parameter to use a later version, I can see 'rack' being 
written to zookeeper.


For now I need to turn my attention to resolving the issues with my 
consumers.


Thanks again for helping out.

Bryan



On 21/09/2018 14:52, Eno Thereska wrote:

Hi Bryan,

I did a simple check with starting a broker with no rack id and then
restarting with a rack id and I can confirm I could get the rack id from
zookeeper after the restart. This was on trunk. Does that basic check work
for you (i.e., without reassigning partitions)?

Thanks
Eno

On Fri, Sep 21, 2018 at 2:07 PM, Bryan Duggan 
wrote:


I didn't get a response to this, but I've been investigating more and can
now frame the problem slightly differently (hopefully, more accurately).

According to this document

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+
data+structures+in+Zookeeper

Which defines broker data structures in zookeeper, the following is the
broker schema (from version 0.10 onwards - I am using version 0.11)

{ "fields":
 [ {"name": "version", "type": "int", "doc": "version id"},
   {"name": "host", "type": "string", "doc": "ip address or host name
of the broker"},
   {"name": "port", "type": "int", "doc": "port of the broker"},
   {"name": "jmx_port", "type": "int", "doc": "port for jmx"}
   {"name": "endpoints", "type": "array", "items": "string", "doc":
"endpoints supported by the broker"}
   {"name": "rack", "type": "string", "doc": "Rack of the broker.
Optional. This will be used in rack aware replication assignment for fault
tolerance."}
 ]
}

when I check my broker data in zookeeper (which has a non-null broker.rack
setting in the properties file), I have the following;

{"endpoints":["PLAINTEXT://x.x.x.x.abcd:9092"],"jmx_port":-1
,"host":"x.x.x.x.abc","timestamp":"1537527988341","port":9092,"version":2}

there is no 'rack'.

In the server.log file on my kafka broker I see;

[2018-09-21 13:00:40,227] INFO KafkaConfig values:
 advertised.host.name = null
 .
 .
 broker.id = 1234567
 broker.rack = rack1
 compression.type = producer
 .
-

so it looks fine from the broker side. However, when I restart kafka on
the host, it doesn't load any rack information into zookeeper.

Can someone please confirm to me, if I have rack awareness, should I
expect to see a value for 'rack' in zookeeper? If so, do I need to do
something else on the broker side to get it to include it as part of the
meta-data it writes (as far as I can see it writes the metadata each time
kafka is restarted).

thanks
Bryan








On 20/09/2018 11:31, Bryan Duggan wrote:


Hi,

I have a kafka cluster consisting of 3 brokers across 3 different AWS
availability zones.  It hosts several topics, each of which has a
replication factor of 3. The cluster is currently not 'rack-aware'.

I am trying to do the following;

 - add 3 additional brokers (one in each of the 3 AZs)

 - make the cluster 'rack-aware'. (ie: create 3 racks on a per-AZ
basic, each containing 2 brokers)

 - reassign the topics with the intention of having 1 replica in each
of the 3 racks.

To achieve this I've added 'broker.rack' to the properties file for each
broker. The rack name is the same as the AZ name each broker is in. I've
restarted kafka on all brokers (in case that's required for rack-awareness
to take effect).

Following restart I've attempted to reassign topics across all 6 brokers
by running the following;

 - ./kafka-reassign-partitions.sh --zookeeper $ZK
--topics-to-move-json-file topics-to-move.json --broker-list '1,2,3,4,5,6'

(where topics-to-move.json is a simple json file containing the topics to
reassign)

The problem I am having is, after running 'kafka-reassign-partitions.sh'
with 6 brokers listed in the broker-list, it doesn't honour
rack-awareness, and instead assigns 2 partitions to brokers in a single
rack with a 3rd being assigned elsewhere.

The version of kafka I am using is 2.11-1.1.1.

Any documentation I've read suggests the above should have achieved what
I want. However, it is not working as expected.

Has anyone else make their kafka cluster 'rack-aware'? If so, did you
experience any issues doing so?

Or, can anyone tell me if there's some step I'm missing to make this work.

TIA

Bryan

Re: Terminate Streams application from within Transformer?

[Bill Bejeck]

Hi Tim,

I wouldn't recommend System.exit(), as it won't give streams a chance to go
through a shutdown process. You can throw a RuntimeException from inside
the Transformer.transform() method and I will bubble up and the Steams
application will shut down.

-Bill

On Fri, Sep 21, 2018 at 6:59 AM Tim Ward  wrote:

> From within a Transformer I detect an error, which is a temporary error,
> such as failure to get a connection to an external microservice.
>
> A typical scenario is that Kuberbetes is starting up a bunch of pods, of
> which mine is one, and they come up in the wrong order. This is perfectly
> normal, and the correct behaviour is for your application to terminate -
> it'll be restarted later, and it should try again, and in due course it'll
> work.
>
> So, inside Transformer.transform() I get one of these errors. I want to
> terminate the application, without of course committing the record
> currently being processed, so that on application restart it re-processes
> this message and everything works fine.
>
> How should I do this? System.exit() or something more sophisticated?
>
> Tim Ward
>
> The contents of this email and any attachment are confidential to the
> intended recipient(s). If you are not an intended recipient: (i) do not
> use, disclose, distribute, copy or publish this email or its contents; (ii)
> please contact the sender immediately; and (iii) delete this email. Our
> privacy policy is available here:
> https://origamienergy.com/privacy-policy/. Origami Energy Limited
> (company number 8619644); Origami Storage Limited (company number 10436515)
> and OSSPV001 Limited (company number 10933403), each registered in England
> and each with a registered office at: Ashcombe Court, Woolsack Way,
> Godalming, GU7 1LQ.
>

Re: Problems trying to make kafka 'rack-aware'

[Eno Thereska]

Hi Bryan,

I did a simple check with starting a broker with no rack id and then
restarting with a rack id and I can confirm I could get the rack id from
zookeeper after the restart. This was on trunk. Does that basic check work
for you (i.e., without reassigning partitions)?

Thanks
Eno

On Fri, Sep 21, 2018 at 2:07 PM, Bryan Duggan 
wrote:

>
> I didn't get a response to this, but I've been investigating more and can
> now frame the problem slightly differently (hopefully, more accurately).
>
> According to this document
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+
> data+structures+in+Zookeeper
>
> Which defines broker data structures in zookeeper, the following is the
> broker schema (from version 0.10 onwards - I am using version 0.11)
>
> { "fields":
> [ {"name": "version", "type": "int", "doc": "version id"},
>   {"name": "host", "type": "string", "doc": "ip address or host name
> of the broker"},
>   {"name": "port", "type": "int", "doc": "port of the broker"},
>   {"name": "jmx_port", "type": "int", "doc": "port for jmx"}
>   {"name": "endpoints", "type": "array", "items": "string", "doc":
> "endpoints supported by the broker"}
>   {"name": "rack", "type": "string", "doc": "Rack of the broker.
> Optional. This will be used in rack aware replication assignment for fault
> tolerance."}
> ]
> }
>
> when I check my broker data in zookeeper (which has a non-null broker.rack
> setting in the properties file), I have the following;
>
> {"endpoints":["PLAINTEXT://x.x.x.x.abcd:9092"],"jmx_port":-1
> ,"host":"x.x.x.x.abc","timestamp":"1537527988341","port":9092,"version":2}
>
> there is no 'rack'.
>
> In the server.log file on my kafka broker I see;
> 
> [2018-09-21 13:00:40,227] INFO KafkaConfig values:
> advertised.host.name = null
> .
> .
> broker.id = 1234567
> broker.rack = rack1
> compression.type = producer
> .
> -
>
> so it looks fine from the broker side. However, when I restart kafka on
> the host, it doesn't load any rack information into zookeeper.
>
> Can someone please confirm to me, if I have rack awareness, should I
> expect to see a value for 'rack' in zookeeper? If so, do I need to do
> something else on the broker side to get it to include it as part of the
> meta-data it writes (as far as I can see it writes the metadata each time
> kafka is restarted).
>
> thanks
> Bryan
>
>
>
>
>
>
>
>
> On 20/09/2018 11:31, Bryan Duggan wrote:
>
>>
>> Hi,
>>
>> I have a kafka cluster consisting of 3 brokers across 3 different AWS
>> availability zones.  It hosts several topics, each of which has a
>> replication factor of 3. The cluster is currently not 'rack-aware'.
>>
>> I am trying to do the following;
>>
>> - add 3 additional brokers (one in each of the 3 AZs)
>>
>> - make the cluster 'rack-aware'. (ie: create 3 racks on a per-AZ
>> basic, each containing 2 brokers)
>>
>> - reassign the topics with the intention of having 1 replica in each
>> of the 3 racks.
>>
>> To achieve this I've added 'broker.rack' to the properties file for each
>> broker. The rack name is the same as the AZ name each broker is in. I've
>> restarted kafka on all brokers (in case that's required for rack-awareness
>> to take effect).
>>
>> Following restart I've attempted to reassign topics across all 6 brokers
>> by running the following;
>>
>> - ./kafka-reassign-partitions.sh --zookeeper $ZK
>> --topics-to-move-json-file topics-to-move.json --broker-list '1,2,3,4,5,6'
>>
>> (where topics-to-move.json is a simple json file containing the topics to
>> reassign)
>>
>> The problem I am having is, after running 'kafka-reassign-partitions.sh'
>> with 6 brokers listed in the broker-list, it doesn't honour
>> rack-awareness, and instead assigns 2 partitions to brokers in a single
>> rack with a 3rd being assigned elsewhere.
>>
>> The version of kafka I am using is 2.11-1.1.1.
>>
>> Any documentation I've read suggests the above should have achieved what
>> I want. However, it is not working as expected.
>>
>> Has anyone else make their kafka cluster 'rack-aware'? If so, did you
>> experience any issues doing so?
>>
>> Or, can anyone tell me if there's some step I'm missing to make this work.
>>
>> TIA
>>
>> Bryan
>>
>>
>>
>>
>

Re: Problems trying to make kafka 'rack-aware'

[Bryan Duggan]

I didn't get a response to this, but I've been investigating more and 
can now frame the problem slightly differently (hopefully, more 
accurately).


According to this document

https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper

Which defines broker data structures in zookeeper, the following is the 
broker schema (from version 0.10 onwards - I am using version 0.11)


{ "fields":
    [ {"name": "version", "type": "int", "doc": "version id"},
  {"name": "host", "type": "string", "doc": "ip address or host 
name of the broker"},

  {"name": "port", "type": "int", "doc": "port of the broker"},
  {"name": "jmx_port", "type": "int", "doc": "port for jmx"}
  {"name": "endpoints", "type": "array", "items": "string", "doc": 
"endpoints supported by the broker"}
  {"name": "rack", "type": "string", "doc": "Rack of the broker. 
Optional. This will be used in rack aware replication assignment for 
fault tolerance."}

    ]
}

when I check my broker data in zookeeper (which has a non-null 
broker.rack setting in the properties file), I have the following;


{"endpoints":["PLAINTEXT://x.x.x.x.abcd:9092"],"jmx_port":-1,"host":"x.x.x.x.abc","timestamp":"1537527988341","port":9092,"version":2}

there is no 'rack'.

In the server.log file on my kafka broker I see;

[2018-09-21 13:00:40,227] INFO KafkaConfig values:
    advertised.host.name = null
    .
    .
    broker.id = 1234567
    broker.rack = rack1
    compression.type = producer
    .
-

so it looks fine from the broker side. However, when I restart kafka on 
the host, it doesn't load any rack information into zookeeper.


Can someone please confirm to me, if I have rack awareness, should I 
expect to see a value for 'rack' in zookeeper? If so, do I need to do 
something else on the broker side to get it to include it as part of the 
meta-data it writes (as far as I can see it writes the metadata each 
time kafka is restarted).


thanks
Bryan







On 20/09/2018 11:31, Bryan Duggan wrote:


Hi,

I have a kafka cluster consisting of 3 brokers across 3 different AWS 
availability zones.  It hosts several topics, each of which has a 
replication factor of 3. The cluster is currently not 'rack-aware'.


I am trying to do the following;

    - add 3 additional brokers (one in each of the 3 AZs)

    - make the cluster 'rack-aware'. (ie: create 3 racks on a per-AZ 
basic, each containing 2 brokers)


    - reassign the topics with the intention of having 1 replica in 
each of the 3 racks.


To achieve this I've added 'broker.rack' to the properties file for 
each broker. The rack name is the same as the AZ name each broker is 
in. I've restarted kafka on all brokers (in case that's required for 
rack-awareness to take effect).


Following restart I've attempted to reassign topics across all 6 
brokers by running the following;


    - ./kafka-reassign-partitions.sh --zookeeper $ZK 
--topics-to-move-json-file topics-to-move.json --broker-list '1,2,3,4,5,6'


(where topics-to-move.json is a simple json file containing the topics 
to reassign)


The problem I am having is, after running 
'kafka-reassign-partitions.sh' with 6 brokers listed in the 
broker-list, it doesn't honour  rack-awareness, and instead assigns 2 
partitions to brokers in a single rack with a 3rd being assigned 
elsewhere.


The version of kafka I am using is 2.11-1.1.1.

Any documentation I've read suggests the above should have achieved 
what I want. However, it is not working as expected.


Has anyone else make their kafka cluster 'rack-aware'? If so, did you 
experience any issues doing so?


Or, can anyone tell me if there's some step I'm missing to make this work.

TIA

Bryan

Re: Kafka consumer offset topic deletion

[Kaushik Nambiar]

Hello,
Please find the server.properties info.
The Kafka server is running on a Ubuntu 14.XX instance.

I have removed all the commented section.
So the data I post below would be the only properties we are using for our
server.
All the other properties are commented(so I guess the default values are
being used).
Also just to add I can see in the log files that topic index segments are
getting deleted at regular intervals.
I am only having issues with the consumer offset topic.

---

broker.id=0

Switch to enable topic deletion or not, default value is false
#delete.topic.enable=true

num.network.threads=3

num.io.threads=8

socket.send.buffer.bytes=102400

socket.receive.buffer.bytes=102400

socket.request.max.bytes=104857600

log.dirs=/tmp/kafka-logs

num.partitions=1

num.recovery.threads.per.data.dir=1

offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

log.retention.hours=168

log.segment.bytes=1073741824 <+11073741824>

log.retention.check.interval.ms=30

zookeeper.connect=localhost:2181

zookeeper.connection.timeout.ms=6000

group.initial.rebalance.delay.ms=0

---

Regards,
KAUSHIK NAMBIAR

On Wed, Sep 19, 2018, 1:55 AM Liam Clarke  wrote:

> Odd that the log compaction isn't working. What OS is your broker running
> on and can you please post your server.properties?
>
> On Wed, 19 Sep. 2018, 2:13 am Kaushik Nambiar,  >
> wrote:
>
> > >
> > > Hello,
> > > We have a Kafka 0.11.xx version setup.
> > > So the system topic which is __consumer_offset, we are looking at many
> > > such topics like __consumer_offset-1,2,4.
> > > So one topic in particular,is now having log segments which is
> > > contributing to 5GB of data.
> > > I had a look at our server.properties file but couldnot find a property
> > > called offsets.retention.minutes (found via Google).
> > > So we are assuming Kafka will take the default value for such
> properties.
> > > So based on the default value of offset.retention.minutes property
> which
> > > is I guess 24 hours,but still we can see log segments older than few
> > month.
> > > So my question is ,is it safe to have that consumer_offset-7 topic
> > (having
> > > 5GB of log segments) deleted.Would it affect kafka anyway that would b
> a
> > > problem in production environment.
> > > If deleting the topics is not a safe implementation,can you suggest a
> > > different fix for this?
> > >
> > >
> > > Thanks and Regards,
> > > Kaushik Nambiar
> > >
> >
>

Terminate Streams application from within Transformer?

[Tim Ward]

>From within a Transformer I detect an error, which is a temporary error, such 
>as failure to get a connection to an external microservice.

A typical scenario is that Kuberbetes is starting up a bunch of pods, of which 
mine is one, and they come up in the wrong order. This is perfectly normal, and 
the correct behaviour is for your application to terminate - it'll be restarted 
later, and it should try again, and in due course it'll work.

So, inside Transformer.transform() I get one of these errors. I want to 
terminate the application, without of course committing the record currently 
being processed, so that on application restart it re-processes this message 
and everything works fine.

How should I do this? System.exit() or something more sophisticated?

Tim Ward

The contents of this email and any attachment are confidential to the intended 
recipient(s). If you are not an intended recipient: (i) do not use, disclose, 
distribute, copy or publish this email or its contents; (ii) please contact the 
sender immediately; and (iii) delete this email. Our privacy policy is 
available here: https://origamienergy.com/privacy-policy/. Origami Energy 
Limited (company number 8619644); Origami Storage Limited (company number 
10436515) and OSSPV001 Limited (company number 10933403), each registered in 
England and each with a registered office at: Ashcombe Court, Woolsack Way, 
Godalming, GU7 1LQ.

Re: Kafka producer huge memory usage (leak?)

[Manikumar]

Hi,

In that case, it is not related to KAFKA-4493. I see StackOverFlowException
in the uploaded heapdump.
I am pretty sure this is related to StackOverFlowException. Some objects
are dangling after this exception.
You need to check/find out when this exception is occuring. check your
application server logs for any errors.
try increasing stack size by adding -Xss2048k.

thanks,

On Fri, Sep 21, 2018 at 2:44 PM Shantanu Deshmukh 
wrote:

> Hi Manikumar,
> I am using correct protocol. SSL and truststore is also correctly
> configured and there is a single port which is for SSL. Otherwise it simply
> wouldn't work. Data is getting produced all fine, it's just that producer
> object is consuming massive amount of memory.
>
> On Fri, Sep 21, 2018 at 2:36 PM Manikumar 
> wrote:
>
> > Hi,
> > Instead trying the PR,  make sure you are setting valid security protocol
> > and connecting to valid broker port.
> > also looks for any errors in producer logs.
> >
> > Thanks,
> >
> >
> >
> >
> >
> > On Fri, Sep 21, 2018 at 12:35 PM Shantanu Deshmukh <
> shantanu...@gmail.com>
> > wrote:
> >
> > > Hi Manikumar,
> > >
> > > I checked this issue. There is this patch available.
> > > https://github.com/apache/kafka/pull/2408.patch
> > >
> > > I pulled Kafka 0.10.1.0 from github. Then tried applying this patch.
> But
> > > several places I am getting error that patch doesn't apply.
> > > I am new to git and patching process. Can you guide me here?
> > >
> > > On Wed, Sep 19, 2018 at 1:02 PM Manikumar 
> > > wrote:
> > >
> > > > Similar issue reported here:KAFKA-7304, but on broker side.  maybe
> you
> > > can
> > > > create a JIRA and upload the heap dump for analysis.
> > > >
> > > > On Wed, Sep 19, 2018 at 11:59 AM Shantanu Deshmukh <
> > > shantanu...@gmail.com>
> > > > wrote:
> > > >
> > > > > Any thoughts on this matter? Someone, please help.
> > > > >
> > > > > On Tue, Sep 18, 2018 at 6:05 PM Shantanu Deshmukh <
> > > shantanu...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Additionally, here's the producer config
> > > > > >
> > > > > > kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092
> > > > > > kafka.acks=0
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.key.serializer=org.apache.kafka.common.serialization.StringSerializer
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.value.serializer=org.apache.kafka.common.serialization.StringSerializer
> > > > > > kafka.max.block.ms=1000
> > > > > > kafka.request.timeout.ms=1000
> > > > > > kafka.max.in.flight.requests.per.connection=1
> > > > > > kafka.retries=0
> > > > > > kafka.compression.type=gzip
> > > > > > kafka.security.protocol=SSL
> > > > > >
> > kafka.ssl.truststore.location=/data/kafka/kafka-server-truststore.jks
> > > > > > kafka.ssl.truststore.password=XX
> > > > > > kafka.linger.ms=300
> > > > > > logger.level=INFO
> > > > > >
> > > > > > On Tue, Sep 18, 2018 at 5:36 PM Shantanu Deshmukh <
> > > > shantanu...@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hello,
> > > > > >>
> > > > > >> We have a 3 broker Kafka 0.10.1.0 deployment in production.
> There
> > > are
> > > > > >> some applications which have Kafka Producers embedded in them
> > which
> > > > send
> > > > > >> application logs to a topic. This topic has 10 partitions with
> > > > > replication
> > > > > >> factor of 3.
> > > > > >>
> > > > > >> We are observing that memory usage on some of these application
> > > > servers
> > > > > >> keep shooting through the roof intermittently. After taking
> > heapdump
> > > > we
> > > > > >> found out that top suspects were:
> > > > > >> *-*
> > > > > >>
> > > > > >>
> > > > > >> *org.apache.kafka.common.network.Selector -*
> > > > > >>
> > > > > >> occupies *352,519,104 (24.96%)* bytes. The memory is accumulated
> > in
> > > > one
> > > > > >> instance of *"byte[]"* loaded by *""*.
> > > > > >>
> > > > > >> *org.apache.kafka.common.network.KafkaChannel -*
> > > > > >>
> > > > > >> occupies *352,527,424 (24.96%)* bytes. The memory is accumulated
> > in
> > > > one
> > > > > >> instance of *"byte[]"* loaded by *""*
> > > > > >>
> > > > > >> * - *
> > > > > >>
> > > > > >> Both of these were holding about 352MB of space. 3 such
> instances,
> > > so
> > > > > >> they were consuming about 1.2GB of memory.
> > > > > >>
> > > > > >> Now regarding usage of producers. Not a huge amount of logs are
> > > being
> > > > > >> sent to Kafka cluster. It is about 200 msgs/sec. Only one
> producer
> > > > > object
> > > > > >> is being used throughout application. Async send function is
> used.
> > > > > >>
> > > > > >> What could be the cause of such huge memory usage? Is this some
> > sort
> > > > of
> > > > > >> memory leak in this specific Kafka version?
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>

Re: Kafka_2.12-1.1.0 config

[hacker win7]

bin/kafka-console-producer.sh add --producer.config
your_producer.properties file.
you don't add --producer.config , you still use the default value of
max.request.size.

sarath reddy  于2018年9月21日周五 上午12:27写道：

> ++pushkar
>
> On Thu 20 Sep, 2018, 16:20 sarath reddy,  wrote:
>
> > Hi Team,
> >
> > We are trying to configure Kafka to produce larger messages,
> >
> > Below are the configs:-
> >
> > Server.properties
> >
> > message.max.bytes=1
> > replica.fetch.max.bytes=10001
> >
> > Producer.properties
> >
> > max.request.size=1
> > compression.type=gzip
> >
> > Consumer.properties
> >
> > fetch.message.max.bytes=1
> >
> > When trying to execute a larger file with below command
> > bin/kafka-console-producer.sh --broker-list localhost:9092 --topic
> > metadata_upload < /export/home/Userid/metadata/TopUBP-1_metadata.json
> >
> > Getting below error:-
> >
> > >[2018-09-19 11:20:03,307] ERROR Error when sending message to topic
> > metadata_upload with key: null, value: 2170060 bytes with error:
> > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> > org.apache.kafka.common.errors.RecordTooLargeException: The message is
> > 2170148 bytes when serialized which is larger than the maximum request
> size
> > you have configured with the max.request.size configuration.
> >
> > Thanks,
> > Sarath Reddy.
> >
> >
> >
>

Re: Kafka producer huge memory usage (leak?)

[Shantanu Deshmukh]

Hi Manikumar,
I am using correct protocol. SSL and truststore is also correctly
configured and there is a single port which is for SSL. Otherwise it simply
wouldn't work. Data is getting produced all fine, it's just that producer
object is consuming massive amount of memory.

On Fri, Sep 21, 2018 at 2:36 PM Manikumar  wrote:

> Hi,
> Instead trying the PR,  make sure you are setting valid security protocol
> and connecting to valid broker port.
> also looks for any errors in producer logs.
>
> Thanks,
>
>
>
>
>
> On Fri, Sep 21, 2018 at 12:35 PM Shantanu Deshmukh 
> wrote:
>
> > Hi Manikumar,
> >
> > I checked this issue. There is this patch available.
> > https://github.com/apache/kafka/pull/2408.patch
> >
> > I pulled Kafka 0.10.1.0 from github. Then tried applying this patch. But
> > several places I am getting error that patch doesn't apply.
> > I am new to git and patching process. Can you guide me here?
> >
> > On Wed, Sep 19, 2018 at 1:02 PM Manikumar 
> > wrote:
> >
> > > Similar issue reported here:KAFKA-7304, but on broker side.  maybe you
> > can
> > > create a JIRA and upload the heap dump for analysis.
> > >
> > > On Wed, Sep 19, 2018 at 11:59 AM Shantanu Deshmukh <
> > shantanu...@gmail.com>
> > > wrote:
> > >
> > > > Any thoughts on this matter? Someone, please help.
> > > >
> > > > On Tue, Sep 18, 2018 at 6:05 PM Shantanu Deshmukh <
> > shantanu...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Additionally, here's the producer config
> > > > >
> > > > > kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092
> > > > > kafka.acks=0
> > > > >
> > > >
> > >
> >
> kafka.key.serializer=org.apache.kafka.common.serialization.StringSerializer
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.value.serializer=org.apache.kafka.common.serialization.StringSerializer
> > > > > kafka.max.block.ms=1000
> > > > > kafka.request.timeout.ms=1000
> > > > > kafka.max.in.flight.requests.per.connection=1
> > > > > kafka.retries=0
> > > > > kafka.compression.type=gzip
> > > > > kafka.security.protocol=SSL
> > > > >
> kafka.ssl.truststore.location=/data/kafka/kafka-server-truststore.jks
> > > > > kafka.ssl.truststore.password=XX
> > > > > kafka.linger.ms=300
> > > > > logger.level=INFO
> > > > >
> > > > > On Tue, Sep 18, 2018 at 5:36 PM Shantanu Deshmukh <
> > > shantanu...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Hello,
> > > > >>
> > > > >> We have a 3 broker Kafka 0.10.1.0 deployment in production. There
> > are
> > > > >> some applications which have Kafka Producers embedded in them
> which
> > > send
> > > > >> application logs to a topic. This topic has 10 partitions with
> > > > replication
> > > > >> factor of 3.
> > > > >>
> > > > >> We are observing that memory usage on some of these application
> > > servers
> > > > >> keep shooting through the roof intermittently. After taking
> heapdump
> > > we
> > > > >> found out that top suspects were:
> > > > >> *-*
> > > > >>
> > > > >>
> > > > >> *org.apache.kafka.common.network.Selector -*
> > > > >>
> > > > >> occupies *352,519,104 (24.96%)* bytes. The memory is accumulated
> in
> > > one
> > > > >> instance of *"byte[]"* loaded by *""*.
> > > > >>
> > > > >> *org.apache.kafka.common.network.KafkaChannel -*
> > > > >>
> > > > >> occupies *352,527,424 (24.96%)* bytes. The memory is accumulated
> in
> > > one
> > > > >> instance of *"byte[]"* loaded by *""*
> > > > >>
> > > > >> * - *
> > > > >>
> > > > >> Both of these were holding about 352MB of space. 3 such instances,
> > so
> > > > >> they were consuming about 1.2GB of memory.
> > > > >>
> > > > >> Now regarding usage of producers. Not a huge amount of logs are
> > being
> > > > >> sent to Kafka cluster. It is about 200 msgs/sec. Only one producer
> > > > object
> > > > >> is being used throughout application. Async send function is used.
> > > > >>
> > > > >> What could be the cause of such huge memory usage? Is this some
> sort
> > > of
> > > > >> memory leak in this specific Kafka version?
> > > > >>
> > > > >>
> > > >
> > >
> >
>

Re: Kafka producer huge memory usage (leak?)

[Manikumar]

Hi,
Instead trying the PR,  make sure you are setting valid security protocol
and connecting to valid broker port.
also looks for any errors in producer logs.

Thanks,





On Fri, Sep 21, 2018 at 12:35 PM Shantanu Deshmukh 
wrote:

> Hi Manikumar,
>
> I checked this issue. There is this patch available.
> https://github.com/apache/kafka/pull/2408.patch
>
> I pulled Kafka 0.10.1.0 from github. Then tried applying this patch. But
> several places I am getting error that patch doesn't apply.
> I am new to git and patching process. Can you guide me here?
>
> On Wed, Sep 19, 2018 at 1:02 PM Manikumar 
> wrote:
>
> > Similar issue reported here:KAFKA-7304, but on broker side.  maybe you
> can
> > create a JIRA and upload the heap dump for analysis.
> >
> > On Wed, Sep 19, 2018 at 11:59 AM Shantanu Deshmukh <
> shantanu...@gmail.com>
> > wrote:
> >
> > > Any thoughts on this matter? Someone, please help.
> > >
> > > On Tue, Sep 18, 2018 at 6:05 PM Shantanu Deshmukh <
> shantanu...@gmail.com
> > >
> > > wrote:
> > >
> > > > Additionally, here's the producer config
> > > >
> > > > kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092
> > > > kafka.acks=0
> > > >
> > >
> >
> kafka.key.serializer=org.apache.kafka.common.serialization.StringSerializer
> > > >
> > > >
> > >
> >
> kafka.value.serializer=org.apache.kafka.common.serialization.StringSerializer
> > > > kafka.max.block.ms=1000
> > > > kafka.request.timeout.ms=1000
> > > > kafka.max.in.flight.requests.per.connection=1
> > > > kafka.retries=0
> > > > kafka.compression.type=gzip
> > > > kafka.security.protocol=SSL
> > > > kafka.ssl.truststore.location=/data/kafka/kafka-server-truststore.jks
> > > > kafka.ssl.truststore.password=XX
> > > > kafka.linger.ms=300
> > > > logger.level=INFO
> > > >
> > > > On Tue, Sep 18, 2018 at 5:36 PM Shantanu Deshmukh <
> > shantanu...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> Hello,
> > > >>
> > > >> We have a 3 broker Kafka 0.10.1.0 deployment in production. There
> are
> > > >> some applications which have Kafka Producers embedded in them which
> > send
> > > >> application logs to a topic. This topic has 10 partitions with
> > > replication
> > > >> factor of 3.
> > > >>
> > > >> We are observing that memory usage on some of these application
> > servers
> > > >> keep shooting through the roof intermittently. After taking heapdump
> > we
> > > >> found out that top suspects were:
> > > >> *-*
> > > >>
> > > >>
> > > >> *org.apache.kafka.common.network.Selector -*
> > > >>
> > > >> occupies *352,519,104 (24.96%)* bytes. The memory is accumulated in
> > one
> > > >> instance of *"byte[]"* loaded by *""*.
> > > >>
> > > >> *org.apache.kafka.common.network.KafkaChannel -*
> > > >>
> > > >> occupies *352,527,424 (24.96%)* bytes. The memory is accumulated in
> > one
> > > >> instance of *"byte[]"* loaded by *""*
> > > >>
> > > >> * - *
> > > >>
> > > >> Both of these were holding about 352MB of space. 3 such instances,
> so
> > > >> they were consuming about 1.2GB of memory.
> > > >>
> > > >> Now regarding usage of producers. Not a huge amount of logs are
> being
> > > >> sent to Kafka cluster. It is about 200 msgs/sec. Only one producer
> > > object
> > > >> is being used throughout application. Async send function is used.
> > > >>
> > > >> What could be the cause of such huge memory usage? Is this some sort
> > of
> > > >> memory leak in this specific Kafka version?
> > > >>
> > > >>
> > >
> >
>

Re: Kafka_2.12-1.1.0 config

[Subash Konar]

You can refer this closed issue in github
.

On Thu, Sep 20, 2018 at 9:57 PM sarath reddy  wrote:

> ++pushkar
>
> On Thu 20 Sep, 2018, 16:20 sarath reddy,  wrote:
>
> > Hi Team,
> >
> > We are trying to configure Kafka to produce larger messages,
> >
> > Below are the configs:-
> >
> > Server.properties
> >
> > message.max.bytes=1
> > replica.fetch.max.bytes=10001
> >
> > Producer.properties
> >
> > max.request.size=1
> > compression.type=gzip
> >
> > Consumer.properties
> >
> > fetch.message.max.bytes=1
> >
> > When trying to execute a larger file with below command
> > bin/kafka-console-producer.sh --broker-list localhost:9092 --topic
> > metadata_upload < /export/home/Userid/metadata/TopUBP-1_metadata.json
> >
> > Getting below error:-
> >
> > >[2018-09-19 11:20:03,307] ERROR Error when sending message to topic
> > metadata_upload with key: null, value: 2170060 bytes with error:
> > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> > org.apache.kafka.common.errors.RecordTooLargeException: The message is
> > 2170148 bytes when serialized which is larger than the maximum request
> size
> > you have configured with the max.request.size configuration.
> >
> > Thanks,
> > Sarath Reddy.
> >
> >
> >
>


-- 
Thanks and Regards,
Subash Konar

Re: Kafka producer huge memory usage (leak?)

[Shantanu Deshmukh]

Hi Manikumar,

I checked this issue. There is this patch available.
https://github.com/apache/kafka/pull/2408.patch

I pulled Kafka 0.10.1.0 from github. Then tried applying this patch. But
several places I am getting error that patch doesn't apply.
I am new to git and patching process. Can you guide me here?

On Wed, Sep 19, 2018 at 1:02 PM Manikumar  wrote:

> Similar issue reported here:KAFKA-7304, but on broker side.  maybe you can
> create a JIRA and upload the heap dump for analysis.
>
> On Wed, Sep 19, 2018 at 11:59 AM Shantanu Deshmukh 
> wrote:
>
> > Any thoughts on this matter? Someone, please help.
> >
> > On Tue, Sep 18, 2018 at 6:05 PM Shantanu Deshmukh  >
> > wrote:
> >
> > > Additionally, here's the producer config
> > >
> > > kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092
> > > kafka.acks=0
> > >
> >
> kafka.key.serializer=org.apache.kafka.common.serialization.StringSerializer
> > >
> > >
> >
> kafka.value.serializer=org.apache.kafka.common.serialization.StringSerializer
> > > kafka.max.block.ms=1000
> > > kafka.request.timeout.ms=1000
> > > kafka.max.in.flight.requests.per.connection=1
> > > kafka.retries=0
> > > kafka.compression.type=gzip
> > > kafka.security.protocol=SSL
> > > kafka.ssl.truststore.location=/data/kafka/kafka-server-truststore.jks
> > > kafka.ssl.truststore.password=XX
> > > kafka.linger.ms=300
> > > logger.level=INFO
> > >
> > > On Tue, Sep 18, 2018 at 5:36 PM Shantanu Deshmukh <
> shantanu...@gmail.com
> > >
> > > wrote:
> > >
> > >> Hello,
> > >>
> > >> We have a 3 broker Kafka 0.10.1.0 deployment in production. There are
> > >> some applications which have Kafka Producers embedded in them which
> send
> > >> application logs to a topic. This topic has 10 partitions with
> > replication
> > >> factor of 3.
> > >>
> > >> We are observing that memory usage on some of these application
> servers
> > >> keep shooting through the roof intermittently. After taking heapdump
> we
> > >> found out that top suspects were:
> > >> *-*
> > >>
> > >>
> > >> *org.apache.kafka.common.network.Selector -*
> > >>
> > >> occupies *352,519,104 (24.96%)* bytes. The memory is accumulated in
> one
> > >> instance of *"byte[]"* loaded by *""*.
> > >>
> > >> *org.apache.kafka.common.network.KafkaChannel -*
> > >>
> > >> occupies *352,527,424 (24.96%)* bytes. The memory is accumulated in
> one
> > >> instance of *"byte[]"* loaded by *""*
> > >>
> > >> * - *
> > >>
> > >> Both of these were holding about 352MB of space. 3 such instances, so
> > >> they were consuming about 1.2GB of memory.
> > >>
> > >> Now regarding usage of producers. Not a huge amount of logs are being
> > >> sent to Kafka cluster. It is about 200 msgs/sec. Only one producer
> > object
> > >> is being used throughout application. Async send function is used.
> > >>
> > >> What could be the cause of such huge memory usage? Is this some sort
> of
> > >> memory leak in this specific Kafka version?
> > >>
> > >>
> >
>