Re: [DISCUSS] PIP-393: Improve performance of Negative Acknowledgement
On 2024/11/26 10:03:06 WenZhi Feng wrote: > I am not specialized in shading dependency of the pulsar, it will be great if > you can help to accomplish this pip together. Regarding the shading configuration, I have cleaned up the existing shading configuration in https://github.com/apache/pulsar/pull/23647 and addressed multiple existing problems. It will be easier to work on the shading config for fastutil after that change has been merged. Please help review https://github.com/apache/pulsar/pull/23647. -Lari
Re: [DISCUSS] PIP-393: Improve performance of Negative Acknowledgement
It's fine to split the implementation in multiple PRs. One detail about the implementation PR title. Please don't add "[pip]" in the PR title of PIP implementations. That prefix is meant to be used for the PIP document PR. -Lari On Tue, 26 Nov 2024 at 12:03, WenZhi Feng wrote: > > Hi Lari. > Thanks for feedback. It is necessary to reduce the size of dependency which > has been added in high level design of the pip. > > > Would it be possible to cover this important aspect as part of PIP-393 > > before taking it to voting? > Do you means that we fix this problem as part of the implementation of > PIP-393? For example, [improve][pip] PIP-393: Improve performance of Negative > Acknowledgement - PART 1 > [improve][pip] PIP-393: Improve performance of Negative Acknowledgement - > PART 2 > part one for original implementation, part two for shading dependency. > > I am not specialized in shading dependency of the pulsar, it will be great if > you can help to accomplish this pip together. > > Thanks. > Wenzhi Feng(thetumbled) > > On 2024/11/26 09:01:59 Lari Hotari wrote: > > Great work, Wenzhi. > > > > The PIP-393 document should include the high level plan of avoiding to > > increase the size of the Pulsar client by the size of fastutil jar > > file. The fastutil jar file is very large, 23MB. We use only a few > > classes of fastutil. There's fastutil-core library which is smaller, > > about ≅6MB. However, that is also relatively large and using > > fastutil-core will introduce another problem on the broker side since > > there's already fastutil jar which also includes fastutil-core jar > > classes. It's necessary to design a proper shading solution as part of > > this PIP design and implementation. > > > > Would it be possible to cover this important aspect as part of PIP-393 > > before taking it to voting? > > > > -Lari > > > > On Fri, 15 Nov 2024 at 10:47, thetumbled wrote: > > > > > > Hi, Pulsar Community. > > > I open a new PIP to fix several issues with negative ack feature. > > > Most importantly, the memory occupation is less than 1% of the original > > > implementation, which make it practical to work in the production > > > enviroment. > > > Some experiment data is listed in the PR, you can also verify the > > > improvement with the test code i show. > > > link: https://github.com/apache/pulsar/pull/23601 > > > > > > Thanks, > > > Wenzhi Feng(thetumbled) > >
Re: [DISCUSS] PIP-393: Improve performance of Negative Acknowledgement
Hi Lari. Thanks for feedback. It is necessary to reduce the size of dependency which has been added in high level design of the pip. > Would it be possible to cover this important aspect as part of PIP-393 > before taking it to voting? Do you means that we fix this problem as part of the implementation of PIP-393? For example, [improve][pip] PIP-393: Improve performance of Negative Acknowledgement - PART 1 [improve][pip] PIP-393: Improve performance of Negative Acknowledgement - PART 2 part one for original implementation, part two for shading dependency. I am not specialized in shading dependency of the pulsar, it will be great if you can help to accomplish this pip together. Thanks. Wenzhi Feng(thetumbled) On 2024/11/26 09:01:59 Lari Hotari wrote: > Great work, Wenzhi. > > The PIP-393 document should include the high level plan of avoiding to > increase the size of the Pulsar client by the size of fastutil jar > file. The fastutil jar file is very large, 23MB. We use only a few > classes of fastutil. There's fastutil-core library which is smaller, > about ≅6MB. However, that is also relatively large and using > fastutil-core will introduce another problem on the broker side since > there's already fastutil jar which also includes fastutil-core jar > classes. It's necessary to design a proper shading solution as part of > this PIP design and implementation. > > Would it be possible to cover this important aspect as part of PIP-393 > before taking it to voting? > > -Lari > > On Fri, 15 Nov 2024 at 10:47, thetumbled wrote: > > > > Hi, Pulsar Community. > > I open a new PIP to fix several issues with negative ack feature. Most > > importantly, the memory occupation is less than 1% of the original > > implementation, which make it practical to work in the production > > enviroment. > > Some experiment data is listed in the PR, you can also verify the > > improvement with the test code i show. > > link: https://github.com/apache/pulsar/pull/23601 > > > > Thanks, > > Wenzhi Feng(thetumbled) >
Re: [DISCUSS] PIP-393: Improve performance of Negative Acknowledgement
Great work, Wenzhi. The PIP-393 document should include the high level plan of avoiding to increase the size of the Pulsar client by the size of fastutil jar file. The fastutil jar file is very large, 23MB. We use only a few classes of fastutil. There's fastutil-core library which is smaller, about ≅6MB. However, that is also relatively large and using fastutil-core will introduce another problem on the broker side since there's already fastutil jar which also includes fastutil-core jar classes. It's necessary to design a proper shading solution as part of this PIP design and implementation. Would it be possible to cover this important aspect as part of PIP-393 before taking it to voting? -Lari On Fri, 15 Nov 2024 at 10:47, thetumbled wrote: > > Hi, Pulsar Community. > I open a new PIP to fix several issues with negative ack feature. Most > importantly, the memory occupation is less than 1% of the original > implementation, which make it practical to work in the production enviroment. > Some experiment data is listed in the PR, you can also verify the > improvement with the test code i show. > link: https://github.com/apache/pulsar/pull/23601 > > Thanks, > Wenzhi Feng(thetumbled)
Re: [DISCUSS] PIP-393: Improve performance of Negative Acknowledgement
This is a very nice improvement and well described No comments. I will be +1 on VOTE Enrico Il giorno mar 26 nov 2024 alle ore 05:12 WenZhi Feng ha scritto: > Hello, Pulsar Community. > I add the space complexity analysis of the new data structure in the PIP: > https://github.com/apache/pulsar/pull/23601. > Before this PIP, the space complexity is 48N byte in best case, 213N byte > in worst case, where N is the number of entries. > After this PIP, the space complexity is 0.163N byte in best case( > 0.163/48=0.3% of the origrinal), 4N byte in worst case(4/213=1.8% of the > origrinal), which is a huge improvement! > Feel free to leave your comments and push this improvement into the pulsar. > > Thanks, > Wenzhi Feng.(thetumbled) > > On 2024/11/15 08:47:33 thetumbled wrote: > > Hi, Pulsar Community. > > I open a new PIP to fix several issues with negative ack feature. > Most importantly, the memory occupation is less than 1% of the original > implementation, which make it practical to work in the production > enviroment. > > Some experiment data is listed in the PR, you can also verify the > improvement with the test code i show. > > link: https://github.com/apache/pulsar/pull/23601 > > > > Thanks, > > Wenzhi Feng(thetumbled) >
Re: [DISCUSS] PIP-393: Improve performance of Negative Acknowledgement
Hello, Pulsar Community. I add the space complexity analysis of the new data structure in the PIP: https://github.com/apache/pulsar/pull/23601. Before this PIP, the space complexity is 48N byte in best case, 213N byte in worst case, where N is the number of entries. After this PIP, the space complexity is 0.163N byte in best case( 0.163/48=0.3% of the origrinal), 4N byte in worst case(4/213=1.8% of the origrinal), which is a huge improvement! Feel free to leave your comments and push this improvement into the pulsar. Thanks, Wenzhi Feng.(thetumbled) On 2024/11/15 08:47:33 thetumbled wrote: > Hi, Pulsar Community. > I open a new PIP to fix several issues with negative ack feature. Most > importantly, the memory occupation is less than 1% of the original > implementation, which make it practical to work in the production enviroment. > Some experiment data is listed in the PR, you can also verify the > improvement with the test code i show. > link: https://github.com/apache/pulsar/pull/23601 > > Thanks, > Wenzhi Feng(thetumbled)
