Re: [Gluster-devel] Multithreaded Iterative Dir Tree Scan

[sankarshan]

On Fri, 24 Apr 2020 at 08:05, Amar Tumballi  wrote:

> This looks like a good effort to pick up Barak. A needed one indeed.
>
>
Should this be tracked with a release label and planned? The content of the
document should probably transfer itself to the issue tracking the PR(s)



> On Mon, Mar 23, 2020 at 3:18 PM Barak Sason Rofman 
> wrote:
>
>> Hello everyone!
>> Following a discussion I had with @Susant Palai some time ago, we have
>> decided to look into an option to improve the rebalance process in the DHT
>> layer by modifying the underlying mechanism. Currently, dir-tree crawling
>> is done recursively, by a single thread, which is likely slow and also
>> poses the risk of stack overflow. An iterative multithreaded solution might
>> improve performance and also stability (by eliminating the risk of stack
>> overflow). I have prepared a POC doc on the matter, including a sample
>> implementation of the iterative multithreaded solution. The doc can be
>> found at:
>>
>> https://docs.google.com/document/d/1JCl0T9zeagOcFFpgVQF8zNyhlR54VqkNAZ7TJb42egE/edit
>>
>> Apart
>> from the rebalance process, maybe this approach can be useful for other
>> use-cases where dir-tree crawl is being performed? Any comments on the
>> concept, the design of the solution and the implementation are welcome.
>>
>> --
>> *Barak Sason Rofman*
>>
>> Gluster Storage Development
>>
>> Red Hat Israel 
>>
>> 34 Jerusalem rd. Ra'anana, 43501
>>
>> bsaso...@redhat.com T: *+972-9-7692304*
>> M: *+972-52-4326355*
>>
>

-- 
sankarshan mukhopadhyay

___

Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968




Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Multithreaded Iterative Dir Tree Scan

[Amar Tumballi]

This looks like a good effort to pick up Barak. A needed one indeed.

-Amar

On Mon, Mar 23, 2020 at 3:18 PM Barak Sason Rofman 
wrote:

> Hello everyone!
> Following a discussion I had with @Susant Palai some time ago, we have
> decided to look into an option to improve the rebalance process in the DHT
> layer by modifying the underlying mechanism. Currently, dir-tree crawling
> is done recursively, by a single thread, which is likely slow and also
> poses the risk of stack overflow. An iterative multithreaded solution might
> improve performance and also stability (by eliminating the risk of stack
> overflow). I have prepared a POC doc on the matter, including a sample
> implementation of the iterative multithreaded solution. The doc can be
> found at:
>
> https://docs.google.com/document/d/1JCl0T9zeagOcFFpgVQF8zNyhlR54VqkNAZ7TJb42egE/edit
>
> Apart
> from the rebalance process, maybe this approach can be useful for other
> use-cases where dir-tree crawl is being performed? Any comments on the
> concept, the design of the solution and the implementation are welcome.
>
> --
> *Barak Sason Rofman*
>
> Gluster Storage Development
>
> Red Hat Israel 
>
> 34 Jerusalem rd. Ra'anana, 43501
>
> bsaso...@redhat.com T: *+972-9-7692304*
> M: *+972-52-4326355*
> 
> ___
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
>
>
>
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>

-- 
--
https://kadalu.io
Container Storage made easy!
___

Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968




Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Multithreaded Iterative Dir Tree Scan

[Barak Sason Rofman]

Hello everyone!
Following a discussion I had with @Susant Palai some time ago, we have
decided to look into an option to improve the rebalance process in the DHT
layer by modifying the underlying mechanism. Currently, dir-tree crawling
is done recursively, by a single thread, which is likely slow and also
poses the risk of stack overflow. An iterative multithreaded solution might
improve performance and also stability (by eliminating the risk of stack
overflow). I have prepared a POC doc on the matter, including a sample
implementation of the iterative multithreaded solution. The doc can be
found at:
https://docs.google.com/document/d/1JCl0T9zeagOcFFpgVQF8zNyhlR54VqkNAZ7TJb42egE/edit
Apart
from the rebalance process, maybe this approach can be useful for other
use-cases where dir-tree crawl is being performed? Any comments on the
concept, the design of the solution and the implementation are welcome.

-- 
*Barak Sason Rofman*

Gluster Storage Development

Red Hat Israel 

34 Jerusalem rd. Ra'anana, 43501

bsaso...@redhat.com T: *+972-9-7692304*
M: *+972-52-4326355*

___

Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968




Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel