Thanks Guozhang and others for helping me. I am able to send and receive
10MB size messages now. In case anybody has the same requirement. Please
make the following changes
*server.properties*
message.max.bytes=10485800
replica.fetch.max.bytes=10485800
socket.send.buffer.bytes=104857600
I think you are printing the class Message instead of MessageAndMetadata.
The output you got was from Message.toString.
Can you just try something like below?
...
ConsumeIterator iter = consumerStream.iterator(); // assuming you have got
a consumer stream.
MessageAndMetadata messageAndMetadta =
Thank you so much Jiangle. I got it working.
I have another problem the consumer doesnt receive message if it is big:
When the producer sends 256kb messages to broker, consumer is able to
retrieve it, but when producer sends 10MB messages to the broker, the
consumer doesn’t receive any message.
Hi Sandeep,
If you are sending messages to different topics, each topic will stick to
a random partition for 10 min. Since they are likely sticking to different
brokers, you will still see messages roughly evenly distributed.
If you are using high level consumer (ZookeeperConsumerConnector),
Hi Jiangjie,
I am using the high level consumer (ZookeeperConsumerConnector), after
getting the message from stream, but I don't see this message.Partition.
Please help me how to get the partition id form message.
What is that I can to do get messages evenly distributed among partitions?
do you
Hi Sandeep,
For old producer, I don’t think you can achieve strict even distribution
of messages across partitions within the same topic. But You can
potentially reduce the sticking time by setting
topic.metadata.refresh.interval.ms to be lower, e.g. 1 second.
Kafka-544 added the partition
Hi Jiangle,
Thanks for the information. This is what I get when I print
MessageandMetadata
*Thread 0: Message(magic = 0, attributes = 0, crc = 127991357, key =
java.nio.HeapByteBuffer[pos=0 lim=1 cap=55], payload =
java.nio.HeapByteBuffer[pos=0 lim=50 cap=50])*
Can you please tell me where can
Dear Developers,
I am using the default partitioning logic(Random Partitioning) to produce
messages into brokers. That is I don't use a partitioner.class.
My requirement is If I produce 10 messages using the below code for a
broker that has 8 partitions across 8 nodes. How many messages will
Palur,
If the 8 partitions are hosted on each one of the nodes, assuming
replication factor 1 then each node will get roughly 10 / 8 messages
due to the random partitioner. If you want to know exactly how many
messages is on each broker then you can use a simple consumer which allows
you to
Thank you Gouzhang. I dont find 0.9 version in the following page
https://www.apache.org/dyn/closer.cgi?path=/kafka/0.8.1.1
I am looking for kafka jar version 0.9.
Can you also tell me how to turn off flushing messages to disk in kafka.
I never want this to happen
Thank you
On Tue, Nov 25,
Palur,
Just adding to what Guozhang said, the answer to your question might
depend on which producer you are using.
Assuming you are producing messages without keys to the same topic, in new
producer(KafkaProducer), the messages will go to brokers in a round robin
way, so the messages will end up
Hi Jiangjie,
This is what I have understood. Please correct me if I am wrong
I don’t use the partition class at all(KeyedMessageString,String data =
new KeyedMessageString, String(topic_name,new_mes). It partitions
messages randomly to different partitions. I don’t see it sticking to any
broker
12 matches
Mail list logo
