Thank you for your reply.
It is about 60 MB.
The size is not huge. But the upload speed is too slow.
No, I did not change "nimbus.thrift.max_buffer_size".
The "getMaxBufferSize" method uses a "nimbus.thrift.max_buffer_size".
So I thought the "getMaxBufferSize" method only impacts a size check in
"stringLengthLimit_" and "maxLength" in the thrift frame.
There are two upload processes when submitting a topology.
first: client -> nimbus
second: nimbus -> supervisor
The second(nimbus -> supervisor) is also slow. However, the upload speed is
faster after I increase "storm.blobstore.inputstream.buffer.size.bytes" to
4MB.
I changed the code to increase the "THRIFT_CHUNK_SIZE_BYTES" value.
client upload speed is faster after I increase "THRIFT_CHUNK_SIZE_BYTES" to
4MB.
So I thought maybe the client should change the "THRIFT_CHUNK_SIZE_BYTES"
like nimbus ("storm.blobstore.inputstream.buffer.size.bytes" value).
Thanks.
2023년 2월 6일 (월) 오후 9:51, Rui Abreu <rui.ab...@gmail.com>님이 작성:
> How big the uber jar that you are submitting?
> Have you tried tweaking the Nimbus side property?
> https://github.com/apache/storm/blob/master/conf/defaults.yaml#L67
>
> On Mon, 6 Feb 2023 at 04:14, Minwoo Kang <its.minwoo.k...@gmail.com>
> wrote:
>
>> Hello, Users.
>>
>> When I submit storm topology, Occasionally it takes a long time.
>> So I investigated that, and I found THRIFT_CHUNK_SIZE_BYTES[1] is too
>> small.
>>
>> However, That value can not be modified because it is the final value.
>> It can be changed (STORM-241[2]) for the first time. But 70174d[3] commit
>> changed chunk size to private constant.
>>
>> I wonder why the user cannot change THRIFT_CHUNK_SIZE_BYTES.
>>
>> Thanks.
>>
>> [1]:
>> https://github.com/apache/storm/blob/bf29d1cc9914d4fe596b5e65532322e3dfd3e4ff/storm-client/src/jvm/org/apache/storm/StormSubmitter.java#L62
>> [2]: https://issues.apache.org/jira/browse/STORM-241
>> [3]:
>> https://github.com/apache/storm/commit/70174deaa758fc1e3f3a881445e8750ae7218b3d
>>
>
