setting max.request.size worked when i did it in producer code; don't understand what producer.properties is for, seems like it's not used.
On Thu, Aug 15, 2019 at 2:25 PM Jonathan Santilli < jonathansanti...@gmail.com> wrote: > I have asked because I did not see that in your previous email when you > tried the console producer. > > Jonathan. > > On Thu, Aug 15, 2019, 3:07 PM l vic <lvic4...@gmail.com> wrote: > > > yes, in producer.properties > > > > On Thu, Aug 15, 2019 at 9:59 AM Jonathan Santilli < > > jonathansanti...@gmail.com> wrote: > > > > > Just to be sure, please confirm the configuration parameter is well > > > set/configure at producer level: > > > > > > max.request.size = 177772390 (for instance) > > > > > > Cheers! > > > -- > > > Jonathan > > > > > > > > > > > > > > > On Thu, Aug 15, 2019 at 1:44 PM l vic <lvic4...@gmail.com> wrote: > > > > > > > I tested it with kafka-console-consumer and kafka-console-producer > > > reading > > > > from large text file (no newlines): > > > > > > > > kafka-console-producer.sh --broker-list 10.10.105.24:6667 --topic > > > test < > > > > ./large-file > > > > > > > > > > > > > > > > > > > > On Thu, Aug 15, 2019 at 4:49 AM Jonathan Santilli < > > > > jonathansanti...@gmail.com> wrote: > > > > > > > > > Hello, try to send and flush just one message of 16777239 bytes, to > > > > verify > > > > > the error still shows up. > > > > > > > > > > Cheers! > > > > > -- > > > > > Jonathan > > > > > > > > > > > > > > > > > > > > On Thu, Aug 15, 2019 at 2:23 AM l vic <lvic4...@gmail.com> wrote: > > > > > > > > > > > My kafka (1.0.0) producer errors out on large (16M) messages. > > > > > > ERROR Error when sending message to topic test with key: null, > > value: > > > > > > 16777239 bytes with error: > > > > (org.apache.kafka.clients.producer.internals. > > > > > > ErrorLoggingCallback) > > > > > > > > > > > > org.apache.kafka.common.errors.RecordTooLargeException: The > message > > > is > > > > > > 16777327 bytes when serialized which is larger than the maximum > > > request > > > > > > size you have configured with the max.request.size configuration. > > > > > > I found couple of links describing the solution: > > > > > > * > > > > > > > > > > > > > > > > > > > > > https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb > > > > > > < > > > > > > > > > > > > > > > > > > > > > https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb > > > > > > >* > > > > > > > > > > > > in my server.properties on brokers I set: > > > > > > socket.request.max.bytes=104857600 > > > > > > message.max.bytes=18874368 > > > > > > max.request.size=18874368 > > > > > > replica.fetch.max.bytes=18874368 > > > > > > fetch.message.max.bytes=18874368 > > > > > > > > > > > > Then in my producer.properties i tried to set > > > > > > max.request.size=18874368 > > > > > > > > > > > > But no matter how i large i try to set max.request.size - > > > > > > i still have the same problem...Are there other settings i am > > > missing? > > > > > > Can it be solved in configuration alone, or do i need to make > code > > > > > changes? > > > > > > Thank you, > > > > > > > > > > > > > > > > > > > > > -- > > > > > Santilli Jonathan > > > > > > > > > > > > > > > > > > -- > > > Santilli Jonathan > > > > > >
