Hi Aaron,
Works wonderfully, thanks ! I've had some issues when using a mask and
errors related to ZSTD compression, but no idea if it is linked to your
code, and no time today to follow up.
Moritz
Am 13.09.2021 17:43 schrieb Aaron Saw Min Sern:
Hi Moritz,
Yes, you can just apply the PR, either by checking into the branch, or
on your local machine, you can rebase the branch on top of the current
main branch (I don't think there will be any conflicts) Let me know if
there are any issues :)
Warm regards,
Aaron
-------------------------
FROM: Moritz Lennert <mlenn...@club.worldonline.be>
SENT: Monday, 13 September 2021, 23:33
TO: Aaron Saw Min Sern
CC: grass-dev@lists.osgeo.org
SUBJECT: Re: [GRASS-dev] [SoC] GSoC 2021 - Final Report
[Parallelization of raster modules for GRASS GIS]
- External Email -
Hi Aaron,
I have a use case for which I would like to try the parallelized
r.neighbors. Do I just have to apply PR 1724 to current main to be
able
to do this, or do I have to do something else ?
Moritz
Am 25.08.2021 15:59 schrieb Aaron Saw Min Sern:
> Hi all,
>
> Thanks Vero for spotting the mistakes on the links. The formatting
of
> the links must have gone wrong, but here's the links to the
respective
> PR.
>
> * r.univar - 1634 [1]
> * r.neighbors - 1724 [2]
> * r.mfilter - 1708 [3]
> * r.resamp.filter - 1759 [4]
> * r.resamp.interp - 1771 [5]
> * r.slope.aspect - 1767 [6]
> * r.series - 1776 [7]
> * r.patch - 1782 [8]
>
> I will still be working on getting the checklists completed in the
> next few weeks.
>
> Warmest regards,
> Aaron
>
> -------------------------
>
> FROM: Veronica Andreo <veroand...@gmail.com>
> SENT: Tuesday, August 24, 2021 8:48 PM
> TO: Aaron Saw Min Sern <aaron...@u.nus.edu>
> CC: s...@lists.osgeo.org <s...@lists.osgeo.org>;
> grass-dev@lists.osgeo.org <grass-dev@lists.osgeo.org>
> SUBJECT: Re: [GRASS-dev] [SoC] GSoC 2021 - Final Report
> [Parallelization of raster modules for GRASS GIS]
>
> - External Email -
>
> Dear Aaron,
>
> Thanks for your report and all your work to make GRASS raster
modules
> run faster, we like that :) Huge thanks as well to Vaclav, Huidae
and
> Maris for their commitment to the project and mentoring! You all
did a
> great team work!
>
> One question: are these changes planned to be merged before the
> creation of grass 8 branch or will they remain for 8+? Maybe add a
> milestone to clarify?
>
> One minor observation: all links currently point to the same PR
>
> We hope to keep seeing you around, Aaron!
>
> All the best,
> Vero
>
> El mar, 24 ago 2021 a las 3:18, Aaron Saw Min Sern
> (<aaron...@u.nus.edu>) escribió:
>
>> Hello everyone,
>>
>> Here is my final report for GSoC 2021 project, _Parallelization of
>> Raster Modules for GRASS GIS_.
>>
>> ABSTRACT
>> The goal of this project is to introduce parallelization to
>> existing raster modules in GRASS GIS using OpenMP. This will allow
>> users to take advantage of more cores in their hardware to speed
up
>> the computation time especially for large raster files with large
>> computation cost. The key challenge of this project is to separate
>> the parallelizable components from the sequential part of the
>> modules without introducing too much overhead in terms of memory,
>> disk or computation resources.
>>
>> MILESTONES
>>
>> In total, I have introduced OpenMP support to 8 raster modules in
>> GRASS GIS. The pull requests to each module are as follows:
>>
>> * r.univar - https://github.com/OSGeo/grass/pull/1634 [1] [1]
>> * r.neighbors - https://github.com/OSGeo/grass/pull/ [2] [1]1724
[2]
>> * r.mfilter - https://github.com/OSGeo/grass/pull/ [2] [1]1708 [3]
>> * r.resamp.filter - https://github.com/OSGeo/grass/pull/ [2]
[1]1759
>> [4]
>> * r.resamp.interp - https://github.com/OSGeo/grass/pull/ [2]
[1]1771
>> [5]
>> * r.slope.aspect - https://github.com/OSGeo/grass/pull/ [2]
[1]1767 [6]
>> * r.series - https://github.com/OSGeo/grass/pull/ [2] [1]1776 [7]
>> * r.patch - https://github.com/OSGeo/grass/pull/ [2] [1]1782 [8]
>>
>> Firstly, I have greatly underestimated the complexity of the work.
>> Up to 20 modules were initially proposed at first but after the
>> second week. However, it became clear that we had to cut down on
the
>> number of target modules and focus more on designing the
algorithms.
>> The modules we targeted behave differently as compared to some
>> modules that had received OpenMP support in the past such as
>> _r.sun_. In particular, the modules need to keep the same of
>> behavior of having low memory footprint even after the
>> parallelization, unlike _r.sun_ which loads the entire raster map
>> in-memory.
>>
>> During the first half of the GSoC, with the mentors’ discussion,
>> we have come out with three different approaches to introducing
>> parallel support to _r.neighbors_. After benchmarking their
>> performance and taking account of their memory/disk usage, we
>> decided to settle with the last approach which requires us to add
an
>> extra parameter _memory_ to allow users to adjust their memory
>> consumption. With this approach, we have to allow the modules to
>> process the raster map by chunks. Once we settled about the
design,
>> we started applying the same approach to other similar modules
with
>> low memory footprints.
>>
>> For more information regarding the implementation, see
>>
https://grasswiki.osgeo.org/wiki/Raster_Parallelization_with_OpenMP
[3]
>> [9].
>>
>> Furthermore, test scripts were included in the modules to ensure
the
>> consistency of the results. Benchmark scripts were added to allow
>> users to easily benchmark the performance of the parallelization
to
>> monitor the speedup in their own local machine. User
documentations
>> were also modified to include sections detailing how to make use
of
>> the newly added features.
>>
>> FUTURE WORK
>>
>> In the future, more raster modules can be parallelized using
similar
>> approach. Then, we can consider tackling more complex modules such
>> as _r.watershed_ and _r.mapcalc_. Also, we could consider
exploring
>> 3D raster modules as well.
>>
>> Furthermore, when we implement parallelization for _r.univar_, we
>> notice that modules that produce statistics involving arithmetic
can
>> often have floating point discrepancies when dealing with large
>> summation. Because of this, computation using different number of
>> threads will now produce different results due to having different
>> order of arithmetic. One idea would be to introduce _Kahan
Summation
>> algorithm_ to reduce the floating-point discrepancies. However,
this
>> still would not guarantee the consistency of results.
>>
>> PERMANENT LINKS
>>
>> For the project overview, please visit the
>>
>
https://summerofcode.withgoogle.com/dashboard/project/6280792767987712/overview
[4]
>> [10]/.
>> For the project timeline and logs, please visit the
>> https://trac.osgeo.org/grass/wiki/GSoC/2021/RasterParallelization
[5]
>> [11].
>>
>> I would like to huge thanks to Huidae Cho, Vaclav Petras and
Māris
>> Nartišs for their valuable guidance. And I would like to thank
the
>> GRASS community for the valuable feedbacks and support. Lastly, I
>> would like to thank the GSoC team for this opportunity.
>>
>> Thanks all!
>>
>> Warmest regards,
>> Aaron Saw Min Sern
>>
>> _______________________________________________
>> grass-dev mailing list
>> grass-dev@lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/grass-dev [6] [12]
>
>
> Links:
> ------
> [1] https://github.com/OSGeo/grass/pull/1634 [1]
> [2] https://github.com/OSGeo/grass/pull/1724 [7]
> [3] https://github.com/OSGeo/grass/pull/1708 [8]
> [4] https://github.com/OSGeo/grass/pull/1759 [9]
> [5] https://github.com/OSGeo/grass/pull/1771 [10]
> [6] https://github.com/OSGeo/grass/pull/1767 [11]
> [7] https://github.com/OSGeo/grass/pull/1776 [12]
> [8] https://github.com/OSGeo/grass/pull/1782 [13]
> [9]
https://grasswiki.osgeo.org/wiki/Raster_Parallelization_with_OpenMP
[3]
> [10]
>
https://summerofcode.withgoogle.com/dashboard/project/6280792767987712/overview/
[14]
> [11]
https://trac.osgeo.org/grass/wiki/GSoC/2021/RasterParallelization [5]
> [12] https://lists.osgeo.org/mailman/listinfo/grass-dev [6]
>
> _______________________________________________
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/grass-dev [6]
Links:
------
[1] https://github.com/OSGeo/grass/pull/1634
[2] https://github.com/OSGeo/grass/pull/
[3] https://grasswiki.osgeo.org/wiki/Raster_Parallelization_with_OpenMP
[4]
https://summerofcode.withgoogle.com/dashboard/project/6280792767987712/overview
[5] https://trac.osgeo.org/grass/wiki/GSoC/2021/RasterParallelization
[6] https://lists.osgeo.org/mailman/listinfo/grass-dev
[7] https://github.com/OSGeo/grass/pull/1724
[8] https://github.com/OSGeo/grass/pull/1708
[9] https://github.com/OSGeo/grass/pull/1759
[10] https://github.com/OSGeo/grass/pull/1771
[11] https://github.com/OSGeo/grass/pull/1767
[12] https://github.com/OSGeo/grass/pull/1776
[13] https://github.com/OSGeo/grass/pull/1782
[14]
https://summerofcode.withgoogle.com/dashboard/project/6280792767987712/overview/
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev
