Re: [GRASS-user] Global overview of GRASS HPC resources/installations ?

[Massi Alvioli]

in any case, if you have examples in which the cloud is conveniente,
there is a case to write about in a paper, so we should converge on
this one;)


M

2018-05-24 17:40 GMT+02:00 Laura Poggio :
> Hi Massi,
> I can see we live in a quite different "computational" world :-)
> I will try to further answer your questions below. I do agree completely
> that if you have access to a good HPC than cloud providers are probably not
> so needed. If you have a good infrastructure that fits (and even goes
> beyond) your needs and it is well maintained. I also think that from the
> point of view of setting up GRASS the two approaches are not so different.
> And it will be interesting to see the development and comparison of the
> different set-ups.
>
> Laura
>
> On 24 May 2018 at 12:12, Massi Alvioli  wrote:
>>
>> even if you can tile-up your problem - which probably covers 95% of
>> the parallelization one can do in GRASS - you still have
>>
>> the problem instantiate the the cloud machines,
>
>
> Scriptable: once the instance template is ready it takes few seconds to
> launch the machines (even hundreds of them)
>
>> copying data to the multiple instances, gather back the results and
>> patching them into your final results - the order of last two steps is
>> your choice - and
>
>
> This is where I am still exploring the most efficient solution. There are
> storage options to avoid or reduce the needs to copy the data to/from the
> instances
>
>>
>> I expect all of these operations to be much slower going through the cloud
>> than in any other
>> architecture.
>>
>> The overall processing time is what matters, from importing initial
>> data to having the final result available.
>
>
> You are right. However I think it depends if the larger data are on own
> premises or online (e.g. remote sensing images). For example for me it is
> much faster to download an image on a online instance, especially when the
> files are already stored on the provider servers.
>
>>
>> Of course, if the cloud is the only viable possibility of having
>> multiple cores, there is no way out. It is also true that everybody
>> owns a couple of desktop machines with a few tens of computing cores
>> overall ..
>>
>
> To set up a cluster with spare desktops you need to follow IT policies and
> sometimes they are not so easy to adapt. In my opinion, it is also not so
> easy  to set up a proper cluster, with shared storage, backup, faster net
> connections, etc.
>
>
>>
>> M
>>
>>
>> 2018-05-24 9:11 GMT+02:00 Laura Poggio :
>> > Hi Massi,
>> > using multiple single instances of GRASS had advantages (in our
>> > workflow)
>> > when tiling: each tile was sent in its own mapset to a different
>> > instance
>> > for processing.
>> > I am aware that this can be done on HPC locally. However, doing this on
>> > the
>> > cloud had the advantage (for us) to be able to use many more instances
>> > than
>> > the cores available locally.
>> >
>> > I think you are right and I/O operation and concurrent database
>> > operations
>> > will be probably slower, but our workflow focus mainly on raster
>> > operations
>> > and integrated GRASS / R models. If these operations can be tiled, then
>> > there are advantages in doing so on different instances, when one does
>> > not
>> > have access to enough local cores.
>> >
>> > I am trying to tidy up the workflow used to be able to share. And I am
>> > looking forward to see other workflows.
>> >
>> > Thanks
>> >
>> > Laura
>> >
>> > On 23 May 2018 at 21:08, Massi Alvioli  wrote:
>> >>
>> >> Hi Laura,
>> >>
>> >> well, not actually - it does not answer my question. I mean, I am
>> >> pretty sure one can have GRASS up and running on some cloud instance,
>> >> but the point is: when it comes to performance, is that convenient? I
>> >> mean multi-process performance, of course. There is not much point on
>> >> running single GRASS instances, if not for very peculiar applications,
>> >> right? I bet it is not convenient, on any level, either if we look at
>> >> I/O operations, or mapcalc operations, not to talk about concurrent
>> >> database operations ... I might be wrong, of course. But my experience
>> >> with cloud environments and parallel processing were rather
>> >> disappointing. On some un-related problem (I mean, not GRASS-related),
>> >> I tried something here https://doi.org/10.30437/ogrs2016_paper_08,
>> >> with little success. I can't imagine a reason why it should be
>> >> different using GRASS modules, while I found undoubtfully good
>> >> performance on HPC machines.
>> >>
>> >> M
>> >>
>> >> 2018-05-23 16:35 GMT+02:00 Laura Poggio :
>> >> > Hi Massi,
>> >> > we managed to run GRASS on different single-core instances on a cloud
>> >> > provider. It was a bit tricky (initially) to set up the NFS mount
>> >> > points. I
>> >> > am still exploring the different types of storage possible and what
>> 

Re: [GRASS-user] Global overview of GRASS HPC resources/installations ?

[Laura Poggio]

Hi Massi,
I can see we live in a quite different "computational" world :-)
I will try to further answer your questions below. I do agree completely
that if you have access to a good HPC than cloud providers are probably not
so needed. If you have a good infrastructure that fits (and even goes
beyond) your needs and it is well maintained. I also think that from the
point of view of setting up GRASS the two approaches are not so different.
And it will be interesting to see the development and comparison of the
different set-ups.

Laura

On 24 May 2018 at 12:12, Massi Alvioli  wrote:

> even if you can tile-up your problem - which probably covers 95% of
> the parallelization one can do in GRASS - you still have

the problem instantiate the the cloud machines,


Scriptable: once the instance template is ready it takes few seconds to
launch the machines (even hundreds of them)

copying data to the multiple instances, gather back the results and
> patching them into your final results - the order of last two steps is
> your choice - and
>

This is where I am still exploring the most efficient solution. There are
storage options to avoid or reduce the needs to copy the data to/from the
instances


> I expect all of these operations to be much slower going through the cloud
> than in any other
> architecture.

The overall processing time is what matters, from importing initial
> data to having the final result available.


You are right. However I think it depends if the larger data are on own
premises or online (e.g. remote sensing images). For example for me it is
much faster to download an image on a online instance, especially when the
files are already stored on the provider servers.


> Of course, if the cloud is the only viable possibility of having
> multiple cores, there is no way out. It is also true that everybody
> owns a couple of desktop machines with a few tens of computing cores
> overall ..
>
>
To set up a cluster with spare desktops you need to follow IT policies and
sometimes they are not so easy to adapt. In my opinion, it is also not so
easy  to set up a proper cluster, with shared storage, backup, faster net
connections, etc.



> M
>
>
> 2018-05-24 9:11 GMT+02:00 Laura Poggio :
> > Hi Massi,
> > using multiple single instances of GRASS had advantages (in our workflow)
> > when tiling: each tile was sent in its own mapset to a different instance
> > for processing.
> > I am aware that this can be done on HPC locally. However, doing this on
> the
> > cloud had the advantage (for us) to be able to use many more instances
> than
> > the cores available locally.
> >
> > I think you are right and I/O operation and concurrent database
> operations
> > will be probably slower, but our workflow focus mainly on raster
> operations
> > and integrated GRASS / R models. If these operations can be tiled, then
> > there are advantages in doing so on different instances, when one does
> not
> > have access to enough local cores.
> >
> > I am trying to tidy up the workflow used to be able to share. And I am
> > looking forward to see other workflows.
> >
> > Thanks
> >
> > Laura
> >
> > On 23 May 2018 at 21:08, Massi Alvioli  wrote:
> >>
> >> Hi Laura,
> >>
> >> well, not actually - it does not answer my question. I mean, I am
> >> pretty sure one can have GRASS up and running on some cloud instance,
> >> but the point is: when it comes to performance, is that convenient? I
> >> mean multi-process performance, of course. There is not much point on
> >> running single GRASS instances, if not for very peculiar applications,
> >> right? I bet it is not convenient, on any level, either if we look at
> >> I/O operations, or mapcalc operations, not to talk about concurrent
> >> database operations ... I might be wrong, of course. But my experience
> >> with cloud environments and parallel processing were rather
> >> disappointing. On some un-related problem (I mean, not GRASS-related),
> >> I tried something here https://doi.org/10.30437/ogrs2016_paper_08,
> >> with little success. I can't imagine a reason why it should be
> >> different using GRASS modules, while I found undoubtfully good
> >> performance on HPC machines.
> >>
> >> M
> >>
> >> 2018-05-23 16:35 GMT+02:00 Laura Poggio :
> >> > Hi Massi,
> >> > we managed to run GRASS on different single-core instances on a cloud
> >> > provider. It was a bit tricky (initially) to set up the NFS mount
> >> > points. I
> >> > am still exploring the different types of storage possible and what
> >> > would be
> >> > cheaper and more efficient.
> >> >
> >> > I hope this answers your question.
> >> >
> >> > Once the workflow is more stable I hope I will be able to share it
> more
> >> > widely.
> >> >
> >> > Thanks
> >> >
> >> > Laura
> >> >
> >> > On 23 May 2018 at 14:37, Massi Alvioli  wrote:
> >> >>
> >> >> Hi Laura,
> >> >>
> >> >> the effort on cloud 

Re: [GRASS-user] Global overview of GRASS HPC resources/installations ?

[Massi Alvioli]

even if you can tile-up your problem - which probably covers 95% of
the parallelization
one can do in GRASS - you still have the problem instantiate the the
cloud machines,
copying data to the multiple instances, gather back the results and
patching them into
your final results - the order of last two steps is your choice - and
I expect all of these
operations to be much slower going through the cloud than in any other
architecture.
The overall processing time is what matters, from importing initial
data to having the
final result available. Of course, if the cloud is the only viable
possibility of having
multiple cores, there is no way out. It is also true that everybody
owns a couple of
desktop machines with a few tens of computing cores overall ..


M


2018-05-24 9:11 GMT+02:00 Laura Poggio :
> Hi Massi,
> using multiple single instances of GRASS had advantages (in our workflow)
> when tiling: each tile was sent in its own mapset to a different instance
> for processing.
> I am aware that this can be done on HPC locally. However, doing this on the
> cloud had the advantage (for us) to be able to use many more instances than
> the cores available locally.
>
> I think you are right and I/O operation and concurrent database operations
> will be probably slower, but our workflow focus mainly on raster operations
> and integrated GRASS / R models. If these operations can be tiled, then
> there are advantages in doing so on different instances, when one does not
> have access to enough local cores.
>
> I am trying to tidy up the workflow used to be able to share. And I am
> looking forward to see other workflows.
>
> Thanks
>
> Laura
>
> On 23 May 2018 at 21:08, Massi Alvioli  wrote:
>>
>> Hi Laura,
>>
>> well, not actually - it does not answer my question. I mean, I am
>> pretty sure one can have GRASS up and running on some cloud instance,
>> but the point is: when it comes to performance, is that convenient? I
>> mean multi-process performance, of course. There is not much point on
>> running single GRASS instances, if not for very peculiar applications,
>> right? I bet it is not convenient, on any level, either if we look at
>> I/O operations, or mapcalc operations, not to talk about concurrent
>> database operations ... I might be wrong, of course. But my experience
>> with cloud environments and parallel processing were rather
>> disappointing. On some un-related problem (I mean, not GRASS-related),
>> I tried something here https://doi.org/10.30437/ogrs2016_paper_08,
>> with little success. I can't imagine a reason why it should be
>> different using GRASS modules, while I found undoubtfully good
>> performance on HPC machines.
>>
>> M
>>
>> 2018-05-23 16:35 GMT+02:00 Laura Poggio :
>> > Hi Massi,
>> > we managed to run GRASS on different single-core instances on a cloud
>> > provider. It was a bit tricky (initially) to set up the NFS mount
>> > points. I
>> > am still exploring the different types of storage possible and what
>> > would be
>> > cheaper and more efficient.
>> >
>> > I hope this answers your question.
>> >
>> > Once the workflow is more stable I hope I will be able to share it more
>> > widely.
>> >
>> > Thanks
>> >
>> > Laura
>> >
>> > On 23 May 2018 at 14:37, Massi Alvioli  wrote:
>> >>
>> >> Hi Laura,
>> >>
>> >> the effort on cloud providers is probably useless. Was it different in
>> >> your case?
>> >>
>> >>
>> >> M
>> >>
>> >> 2018-05-22 10:12 GMT+02:00 Laura Poggio :
>> >> > I am really interested in this. I am experimenting with different
>> >> > settings
>> >> > to use GRASS on HPC, more specifically on multi-core local machines
>> >> > and
>> >> > on
>> >> > single-core multiple instances on a cloud provider. It would be great
>> >> > to
>> >> > share experiences with other people fighting the same problems.
>> >> >
>> >> > Thanks
>> >> >
>> >> > Laura
>> >> >
>> >> > On 20 May 2018 at 12:32, Moritz Lennert
>> >> > 
>> >> > wrote:
>> >> >>
>> >> >> Le Sun, 20 May 2018 09:30:53 +0200,
>> >> >> Nikos Alexandris  a écrit :
>> >> >>
>> >> >> > * Massi Alvioli  [2018-05-17 15:01:39 +0200]:
>> >> >> >
>> >> >> > >2018-05-17 10:09 GMT+02:00 Moritz Lennert
>> >> >> > >:
>> >> >> > >
>> >> >> > >Hi,
>> >> >> > >
>> >> >> > >> [I imagine your mail was supposed to go onto the mailing list
>> >> >> > >> and
>> >> >> > >> not just to me...]
>> >> >> > >
>> >> >> > >sure my answer was for everyone to read, I believe I tried to
>> >> >> > > send
>> >> >> > > it
>> >> >> > >again afterwards..
>> >> >> > >something must have gone wrong.
>> >> >> > >
>> >> >> > >> I just presented GRASS and a short overview over GRASS on HPC
>> >> >> > >> yesterday at the FOSS4F-FR and there was a lot of interest for
>> >> >> > >> this. Several people asked me about specific documentation on
>> 

Re: [GRASS-user] Global overview of GRASS HPC resources/installations ?

[Laura Poggio]

Hi Massi,
using multiple single instances of GRASS had advantages (in our workflow)
when tiling: each tile was sent in its own mapset to a different instance
for processing.
I am aware that this can be done on HPC locally. However, doing this on the
cloud had the advantage (for us) to be able to use many more instances than
the cores available locally.

I think you are right and I/O operation and concurrent database operations
will be probably slower, but our workflow focus mainly on raster operations
and integrated GRASS / R models. If these operations can be tiled, then
there are advantages in doing so on different instances, when one does not
have access to enough local cores.

I am trying to tidy up the workflow used to be able to share. And I am
looking forward to see other workflows.

Thanks

Laura

On 23 May 2018 at 21:08, Massi Alvioli  wrote:

> Hi Laura,
>
> well, not actually - it does not answer my question. I mean, I am
> pretty sure one can have GRASS up and running on some cloud instance,
> but the point is: when it comes to performance, is that convenient? I
> mean multi-process performance, of course. There is not much point on
> running single GRASS instances, if not for very peculiar applications,
> right? I bet it is not convenient, on any level, either if we look at
> I/O operations, or mapcalc operations, not to talk about concurrent
> database operations ... I might be wrong, of course. But my experience
> with cloud environments and parallel processing were rather
> disappointing. On some un-related problem (I mean, not GRASS-related),
> I tried something here https://doi.org/10.30437/ogrs2016_paper_08,
> with little success. I can't imagine a reason why it should be
> different using GRASS modules, while I found undoubtfully good
> performance on HPC machines.
>
> M
>
> 2018-05-23 16:35 GMT+02:00 Laura Poggio :
> > Hi Massi,
> > we managed to run GRASS on different single-core instances on a cloud
> > provider. It was a bit tricky (initially) to set up the NFS mount
> points. I
> > am still exploring the different types of storage possible and what
> would be
> > cheaper and more efficient.
> >
> > I hope this answers your question.
> >
> > Once the workflow is more stable I hope I will be able to share it more
> > widely.
> >
> > Thanks
> >
> > Laura
> >
> > On 23 May 2018 at 14:37, Massi Alvioli  wrote:
> >>
> >> Hi Laura,
> >>
> >> the effort on cloud providers is probably useless. Was it different in
> >> your case?
> >>
> >>
> >> M
> >>
> >> 2018-05-22 10:12 GMT+02:00 Laura Poggio :
> >> > I am really interested in this. I am experimenting with different
> >> > settings
> >> > to use GRASS on HPC, more specifically on multi-core local machines
> and
> >> > on
> >> > single-core multiple instances on a cloud provider. It would be great
> to
> >> > share experiences with other people fighting the same problems.
> >> >
> >> > Thanks
> >> >
> >> > Laura
> >> >
> >> > On 20 May 2018 at 12:32, Moritz Lennert  >
> >> > wrote:
> >> >>
> >> >> Le Sun, 20 May 2018 09:30:53 +0200,
> >> >> Nikos Alexandris  a écrit :
> >> >>
> >> >> > * Massi Alvioli  [2018-05-17 15:01:39 +0200]:
> >> >> >
> >> >> > >2018-05-17 10:09 GMT+02:00 Moritz Lennert
> >> >> > >:
> >> >> > >
> >> >> > >Hi,
> >> >> > >
> >> >> > >> [I imagine your mail was supposed to go onto the mailing list
> and
> >> >> > >> not just to me...]
> >> >> > >
> >> >> > >sure my answer was for everyone to read, I believe I tried to send
> >> >> > > it
> >> >> > >again afterwards..
> >> >> > >something must have gone wrong.
> >> >> > >
> >> >> > >> I just presented GRASS and a short overview over GRASS on HPC
> >> >> > >> yesterday at the FOSS4F-FR and there was a lot of interest for
> >> >> > >> this. Several people asked me about specific documentation on
> the
> >> >> > >> subject.
> >> >> > >
> >> >> > >What we did about GRASS + HPC was for specific production purposes
> >> >> > >and no documentation
> >> >> > >whatsoever wascreated, basically due to lack of time.. so I find
> it
> >> >> > >hard to say whether this is going
> >> >> > >to change in the near future:). Surely the topic is of wide
> interest
> >> >> > >and worth being discussed in
> >> >> > >several contexts.
> >> >> > >
> >> >> > >> Currently, I'm aware of the following wiki pages which each
> >> >> > >> potentially touches on some aspects of HPC:
> >> >> > >
> >> >> > >I must admit that existing documentation/papers did not help much.
> >> >> > >Well, did not help at all, actually.
> >> >> > >One major problem in my opinion/experience is that
> >> >> > >multi-core/multi-node machines can be really
> >> >> > >different from each other, and parallelization strategies very
> >> >> > >purpose-specific, so that creating
> >> >> > >general-purpose documents/papers, or even 

Re: [GRASS-user] Global overview of GRASS HPC resources/installations ?

[Massi Alvioli]

Hi Laura,

well, not actually - it does not answer my question. I mean, I am
pretty sure one can have GRASS up and running on some cloud instance,
but the point is: when it comes to performance, is that convenient? I
mean multi-process performance, of course. There is not much point on
running single GRASS instances, if not for very peculiar applications,
right? I bet it is not convenient, on any level, either if we look at
I/O operations, or mapcalc operations, not to talk about concurrent
database operations ... I might be wrong, of course. But my experience
with cloud environments and parallel processing were rather
disappointing. On some un-related problem (I mean, not GRASS-related),
I tried something here https://doi.org/10.30437/ogrs2016_paper_08,
with little success. I can't imagine a reason why it should be
different using GRASS modules, while I found undoubtfully good
performance on HPC machines.

M

2018-05-23 16:35 GMT+02:00 Laura Poggio :
> Hi Massi,
> we managed to run GRASS on different single-core instances on a cloud
> provider. It was a bit tricky (initially) to set up the NFS mount points. I
> am still exploring the different types of storage possible and what would be
> cheaper and more efficient.
>
> I hope this answers your question.
>
> Once the workflow is more stable I hope I will be able to share it more
> widely.
>
> Thanks
>
> Laura
>
> On 23 May 2018 at 14:37, Massi Alvioli  wrote:
>>
>> Hi Laura,
>>
>> the effort on cloud providers is probably useless. Was it different in
>> your case?
>>
>>
>> M
>>
>> 2018-05-22 10:12 GMT+02:00 Laura Poggio :
>> > I am really interested in this. I am experimenting with different
>> > settings
>> > to use GRASS on HPC, more specifically on multi-core local machines and
>> > on
>> > single-core multiple instances on a cloud provider. It would be great to
>> > share experiences with other people fighting the same problems.
>> >
>> > Thanks
>> >
>> > Laura
>> >
>> > On 20 May 2018 at 12:32, Moritz Lennert 
>> > wrote:
>> >>
>> >> Le Sun, 20 May 2018 09:30:53 +0200,
>> >> Nikos Alexandris  a écrit :
>> >>
>> >> > * Massi Alvioli  [2018-05-17 15:01:39 +0200]:
>> >> >
>> >> > >2018-05-17 10:09 GMT+02:00 Moritz Lennert
>> >> > >:
>> >> > >
>> >> > >Hi,
>> >> > >
>> >> > >> [I imagine your mail was supposed to go onto the mailing list and
>> >> > >> not just to me...]
>> >> > >
>> >> > >sure my answer was for everyone to read, I believe I tried to send
>> >> > > it
>> >> > >again afterwards..
>> >> > >something must have gone wrong.
>> >> > >
>> >> > >> I just presented GRASS and a short overview over GRASS on HPC
>> >> > >> yesterday at the FOSS4F-FR and there was a lot of interest for
>> >> > >> this. Several people asked me about specific documentation on the
>> >> > >> subject.
>> >> > >
>> >> > >What we did about GRASS + HPC was for specific production purposes
>> >> > >and no documentation
>> >> > >whatsoever wascreated, basically due to lack of time.. so I find it
>> >> > >hard to say whether this is going
>> >> > >to change in the near future:). Surely the topic is of wide interest
>> >> > >and worth being discussed in
>> >> > >several contexts.
>> >> > >
>> >> > >> Currently, I'm aware of the following wiki pages which each
>> >> > >> potentially touches on some aspects of HPC:
>> >> > >
>> >> > >I must admit that existing documentation/papers did not help much.
>> >> > >Well, did not help at all, actually.
>> >> > >One major problem in my opinion/experience is that
>> >> > >multi-core/multi-node machines can be really
>> >> > >different from each other, and parallelization strategies very
>> >> > >purpose-specific, so that creating
>> >> > >general-purpose documents/papers, or even software, *may* be a
>> >> > >hopeless effort. Smart ideas
>> >> > >are most welcome, of course:)
>> >> >
>> >> > Dear Massimo and all,
>> >> >
>> >> > Being a beginner in massively processing Landsat 8 images using JRC's
>> >> > JEODPP system (which is designed for High-Throughput,
>> >> > https://doi.org/10.1016/j.future.2017.11.007), I found useful notes
>> >> > in
>> >> > the Wiki (notably Veronica's excellent tutorials) and elsewhere, got
>> >> > specific answers through the mailing lists and learned a lot in
>> >> > on-site discussions during the last OSGeo sprint, for example.
>> >> >
>> >> > Nonetheless, I think to have learned quite some things the hard way.
>> >> > In this regard, some answers to even "non-sense" questions are worth
>> >> > documenting.
>> >> >
>> >> > My aim is to transfer notes of practical value. Having HPC and HTC
>> >> > related notes in a wiki, will help to get started, promote best
>> >> > practices, learn through common mistakes and give an overview for the
>> >> > points Peter put in this thread's first message.
>> >>
>> >> +1
>> >>
>> >> >
>> >> > I hope 

Re: [GRASS-user] Global overview of GRASS HPC resources/installations ?

[Laura Poggio]

Hi Massi,
we managed to run GRASS on different single-core instances on a cloud
provider. It was a bit tricky (initially) to set up the NFS mount points. I
am still exploring the different types of storage possible and what would
be cheaper and more efficient.

I hope this answers your question.

Once the workflow is more stable I hope I will be able to share it more
widely.

Thanks

Laura

On 23 May 2018 at 14:37, Massi Alvioli  wrote:

> Hi Laura,
>
> the effort on cloud providers is probably useless. Was it different in
> your case?
>
>
> M
>
> 2018-05-22 10:12 GMT+02:00 Laura Poggio :
> > I am really interested in this. I am experimenting with different
> settings
> > to use GRASS on HPC, more specifically on multi-core local machines and
> on
> > single-core multiple instances on a cloud provider. It would be great to
> > share experiences with other people fighting the same problems.
> >
> > Thanks
> >
> > Laura
> >
> > On 20 May 2018 at 12:32, Moritz Lennert 
> > wrote:
> >>
> >> Le Sun, 20 May 2018 09:30:53 +0200,
> >> Nikos Alexandris  a écrit :
> >>
> >> > * Massi Alvioli  [2018-05-17 15:01:39 +0200]:
> >> >
> >> > >2018-05-17 10:09 GMT+02:00 Moritz Lennert
> >> > >:
> >> > >
> >> > >Hi,
> >> > >
> >> > >> [I imagine your mail was supposed to go onto the mailing list and
> >> > >> not just to me...]
> >> > >
> >> > >sure my answer was for everyone to read, I believe I tried to send it
> >> > >again afterwards..
> >> > >something must have gone wrong.
> >> > >
> >> > >> I just presented GRASS and a short overview over GRASS on HPC
> >> > >> yesterday at the FOSS4F-FR and there was a lot of interest for
> >> > >> this. Several people asked me about specific documentation on the
> >> > >> subject.
> >> > >
> >> > >What we did about GRASS + HPC was for specific production purposes
> >> > >and no documentation
> >> > >whatsoever wascreated, basically due to lack of time.. so I find it
> >> > >hard to say whether this is going
> >> > >to change in the near future:). Surely the topic is of wide interest
> >> > >and worth being discussed in
> >> > >several contexts.
> >> > >
> >> > >> Currently, I'm aware of the following wiki pages which each
> >> > >> potentially touches on some aspects of HPC:
> >> > >
> >> > >I must admit that existing documentation/papers did not help much.
> >> > >Well, did not help at all, actually.
> >> > >One major problem in my opinion/experience is that
> >> > >multi-core/multi-node machines can be really
> >> > >different from each other, and parallelization strategies very
> >> > >purpose-specific, so that creating
> >> > >general-purpose documents/papers, or even software, *may* be a
> >> > >hopeless effort. Smart ideas
> >> > >are most welcome, of course:)
> >> >
> >> > Dear Massimo and all,
> >> >
> >> > Being a beginner in massively processing Landsat 8 images using JRC's
> >> > JEODPP system (which is designed for High-Throughput,
> >> > https://doi.org/10.1016/j.future.2017.11.007), I found useful notes
> in
> >> > the Wiki (notably Veronica's excellent tutorials) and elsewhere, got
> >> > specific answers through the mailing lists and learned a lot in
> >> > on-site discussions during the last OSGeo sprint, for example.
> >> >
> >> > Nonetheless, I think to have learned quite some things the hard way.
> >> > In this regard, some answers to even "non-sense" questions are worth
> >> > documenting.
> >> >
> >> > My aim is to transfer notes of practical value. Having HPC and HTC
> >> > related notes in a wiki, will help to get started, promote best
> >> > practices, learn through common mistakes and give an overview for the
> >> > points Peter put in this thread's first message.
> >>
> >> +1
> >>
> >> >
> >> > I hope it's fine to name the page "High Performance Computing". Please
> >> > advise or create a page with another name if you think otherwise.
> >>
> >>
> >> +1
> >>
> >> Moritz
> >> ___
> >> grass-user mailing list
> >> grass-user@lists.osgeo.org
> >> https://lists.osgeo.org/mailman/listinfo/grass-user
> >
> >
> >
> > ___
> > grass-user mailing list
> > grass-user@lists.osgeo.org
> > https://lists.osgeo.org/mailman/listinfo/grass-user
>
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Global overview of GRASS HPC resources/installations ?

[Massi Alvioli]

Hi Laura,

the effort on cloud providers is probably useless. Was it different in
your case?


M

2018-05-22 10:12 GMT+02:00 Laura Poggio :
> I am really interested in this. I am experimenting with different settings
> to use GRASS on HPC, more specifically on multi-core local machines and on
> single-core multiple instances on a cloud provider. It would be great to
> share experiences with other people fighting the same problems.
>
> Thanks
>
> Laura
>
> On 20 May 2018 at 12:32, Moritz Lennert 
> wrote:
>>
>> Le Sun, 20 May 2018 09:30:53 +0200,
>> Nikos Alexandris  a écrit :
>>
>> > * Massi Alvioli  [2018-05-17 15:01:39 +0200]:
>> >
>> > >2018-05-17 10:09 GMT+02:00 Moritz Lennert
>> > >:
>> > >
>> > >Hi,
>> > >
>> > >> [I imagine your mail was supposed to go onto the mailing list and
>> > >> not just to me...]
>> > >
>> > >sure my answer was for everyone to read, I believe I tried to send it
>> > >again afterwards..
>> > >something must have gone wrong.
>> > >
>> > >> I just presented GRASS and a short overview over GRASS on HPC
>> > >> yesterday at the FOSS4F-FR and there was a lot of interest for
>> > >> this. Several people asked me about specific documentation on the
>> > >> subject.
>> > >
>> > >What we did about GRASS + HPC was for specific production purposes
>> > >and no documentation
>> > >whatsoever wascreated, basically due to lack of time.. so I find it
>> > >hard to say whether this is going
>> > >to change in the near future:). Surely the topic is of wide interest
>> > >and worth being discussed in
>> > >several contexts.
>> > >
>> > >> Currently, I'm aware of the following wiki pages which each
>> > >> potentially touches on some aspects of HPC:
>> > >
>> > >I must admit that existing documentation/papers did not help much.
>> > >Well, did not help at all, actually.
>> > >One major problem in my opinion/experience is that
>> > >multi-core/multi-node machines can be really
>> > >different from each other, and parallelization strategies very
>> > >purpose-specific, so that creating
>> > >general-purpose documents/papers, or even software, *may* be a
>> > >hopeless effort. Smart ideas
>> > >are most welcome, of course:)
>> >
>> > Dear Massimo and all,
>> >
>> > Being a beginner in massively processing Landsat 8 images using JRC's
>> > JEODPP system (which is designed for High-Throughput,
>> > https://doi.org/10.1016/j.future.2017.11.007), I found useful notes in
>> > the Wiki (notably Veronica's excellent tutorials) and elsewhere, got
>> > specific answers through the mailing lists and learned a lot in
>> > on-site discussions during the last OSGeo sprint, for example.
>> >
>> > Nonetheless, I think to have learned quite some things the hard way.
>> > In this regard, some answers to even "non-sense" questions are worth
>> > documenting.
>> >
>> > My aim is to transfer notes of practical value. Having HPC and HTC
>> > related notes in a wiki, will help to get started, promote best
>> > practices, learn through common mistakes and give an overview for the
>> > points Peter put in this thread's first message.
>>
>> +1
>>
>> >
>> > I hope it's fine to name the page "High Performance Computing". Please
>> > advise or create a page with another name if you think otherwise.
>>
>>
>> +1
>>
>> Moritz
>> ___
>> grass-user mailing list
>> grass-user@lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/grass-user
>
>
>
> ___
> grass-user mailing list
> grass-user@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/grass-user
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Global overview of GRASS HPC resources/installations ?

[Nikos Alexandris]

Dear all,

as a first step, I collected the links you provided in this thread.
In trying to cover as much as possible, I named the page
https://grasswiki.osgeo.org/wiki/Supercomputing, after Wikipedia's
corresponding article.

Please modify as you see fit and let us enrich this page.

Thank you, Nikos


signature.asc
Description: PGP signature
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Global overview of GRASS HPC resources/installations ?

[Laura Poggio]

I am really interested in this. I am experimenting with different settings
to use GRASS on HPC, more specifically on multi-core local machines and on
single-core multiple instances on a cloud provider. It would be great to
share experiences with other people fighting the same problems.

Thanks

Laura

On 20 May 2018 at 12:32, Moritz Lennert 
wrote:

> Le Sun, 20 May 2018 09:30:53 +0200,
> Nikos Alexandris  a écrit :
>
> > * Massi Alvioli  [2018-05-17 15:01:39 +0200]:
> >
> > >2018-05-17 10:09 GMT+02:00 Moritz Lennert
> > >:
> > >
> > >Hi,
> > >
> > >> [I imagine your mail was supposed to go onto the mailing list and
> > >> not just to me...]
> > >
> > >sure my answer was for everyone to read, I believe I tried to send it
> > >again afterwards..
> > >something must have gone wrong.
> > >
> > >> I just presented GRASS and a short overview over GRASS on HPC
> > >> yesterday at the FOSS4F-FR and there was a lot of interest for
> > >> this. Several people asked me about specific documentation on the
> > >> subject.
> > >
> > >What we did about GRASS + HPC was for specific production purposes
> > >and no documentation
> > >whatsoever wascreated, basically due to lack of time.. so I find it
> > >hard to say whether this is going
> > >to change in the near future:). Surely the topic is of wide interest
> > >and worth being discussed in
> > >several contexts.
> > >
> > >> Currently, I'm aware of the following wiki pages which each
> > >> potentially touches on some aspects of HPC:
> > >
> > >I must admit that existing documentation/papers did not help much.
> > >Well, did not help at all, actually.
> > >One major problem in my opinion/experience is that
> > >multi-core/multi-node machines can be really
> > >different from each other, and parallelization strategies very
> > >purpose-specific, so that creating
> > >general-purpose documents/papers, or even software, *may* be a
> > >hopeless effort. Smart ideas
> > >are most welcome, of course:)
> >
> > Dear Massimo and all,
> >
> > Being a beginner in massively processing Landsat 8 images using JRC's
> > JEODPP system (which is designed for High-Throughput,
> > https://doi.org/10.1016/j.future.2017.11.007), I found useful notes in
> > the Wiki (notably Veronica's excellent tutorials) and elsewhere, got
> > specific answers through the mailing lists and learned a lot in
> > on-site discussions during the last OSGeo sprint, for example.
> >
> > Nonetheless, I think to have learned quite some things the hard way.
> > In this regard, some answers to even "non-sense" questions are worth
> > documenting.
> >
> > My aim is to transfer notes of practical value. Having HPC and HTC
> > related notes in a wiki, will help to get started, promote best
> > practices, learn through common mistakes and give an overview for the
> > points Peter put in this thread's first message.
>
> +1
>
> >
> > I hope it's fine to name the page "High Performance Computing". Please
> > advise or create a page with another name if you think otherwise.
>
>
> +1
>
> Moritz
> ___
> grass-user mailing list
> grass-user@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/grass-user
>
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Global overview of GRASS HPC resources/installations ?

[Moritz Lennert]

Le Sun, 20 May 2018 09:30:53 +0200,
Nikos Alexandris  a écrit :

> * Massi Alvioli  [2018-05-17 15:01:39 +0200]:
> 
> >2018-05-17 10:09 GMT+02:00 Moritz Lennert
> >:
> >
> >Hi,
> >  
> >> [I imagine your mail was supposed to go onto the mailing list and
> >> not just to me...]  
> >
> >sure my answer was for everyone to read, I believe I tried to send it
> >again afterwards..
> >something must have gone wrong.
> >  
> >> I just presented GRASS and a short overview over GRASS on HPC
> >> yesterday at the FOSS4F-FR and there was a lot of interest for
> >> this. Several people asked me about specific documentation on the
> >> subject.  
> >
> >What we did about GRASS + HPC was for specific production purposes
> >and no documentation
> >whatsoever wascreated, basically due to lack of time.. so I find it
> >hard to say whether this is going
> >to change in the near future:). Surely the topic is of wide interest
> >and worth being discussed in
> >several contexts.
> >  
> >> Currently, I'm aware of the following wiki pages which each
> >> potentially touches on some aspects of HPC:  
> >
> >I must admit that existing documentation/papers did not help much.
> >Well, did not help at all, actually.
> >One major problem in my opinion/experience is that
> >multi-core/multi-node machines can be really
> >different from each other, and parallelization strategies very
> >purpose-specific, so that creating
> >general-purpose documents/papers, or even software, *may* be a
> >hopeless effort. Smart ideas
> >are most welcome, of course:)  
> 
> Dear Massimo and all,
> 
> Being a beginner in massively processing Landsat 8 images using JRC's
> JEODPP system (which is designed for High-Throughput,
> https://doi.org/10.1016/j.future.2017.11.007), I found useful notes in
> the Wiki (notably Veronica's excellent tutorials) and elsewhere, got
> specific answers through the mailing lists and learned a lot in
> on-site discussions during the last OSGeo sprint, for example.
> 
> Nonetheless, I think to have learned quite some things the hard way.
> In this regard, some answers to even "non-sense" questions are worth
> documenting.
> 
> My aim is to transfer notes of practical value. Having HPC and HTC
> related notes in a wiki, will help to get started, promote best
> practices, learn through common mistakes and give an overview for the
> points Peter put in this thread's first message.

+1

> 
> I hope it's fine to name the page "High Performance Computing". Please
> advise or create a page with another name if you think otherwise.


+1

Moritz
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Global overview of GRASS HPC resources/installations ?

[Markus Neteler]

On Sun, May 20, 2018 at 9:30 AM, Nikos Alexandris
 wrote:
...
> My aim is to transfer notes of practical value. Having HPC and HTC
> related notes in a wiki, will help to get started, promote best
> practices, learn through common mistakes and give an overview for the
> points Peter put in this thread's first message.

Great!
Happy to contribute as well.

BTW: I still have access to the Slovak supercomputer "Aurel"
https://vs.sav.sk/?lang=en=departments=vvt=services

which is a Power7 architecture. Of course GRASS GIS 7.4 is installed there :)


> I hope it's fine to name the page "High Performance Computing". Please
> advise or create a page with another name if you think otherwise.

Please go ahead :-)

BTW: the other day I got an invitation to contribute to

Special Issue "High-Performance Computing in Geoscience and Remote Sensing"
Deadline for manuscript submissions: 20 September 2018
http://www.mdpi.com/si/sensors/Computing_Geoscience

Perhaps some of us write a joint article?

Markus
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Global overview of GRASS HPC resources/installations ?

[Nikos Alexandris]

* Massi Alvioli  [2018-05-17 15:01:39 +0200]:


2018-05-17 10:09 GMT+02:00 Moritz Lennert :

Hi,


[I imagine your mail was supposed to go onto the mailing list and not just to 
me...]


sure my answer was for everyone to read, I believe I tried to send it
again afterwards..
something must have gone wrong.


I just presented GRASS and a short overview over GRASS on HPC yesterday at
the FOSS4F-FR and there was a lot of interest for this. Several people asked
me about specific documentation on the subject.


What we did about GRASS + HPC was for specific production purposes and
no documentation
whatsoever wascreated, basically due to lack of time.. so I find it
hard to say whether this is going
to change in the near future:). Surely the topic is of wide interest
and worth being discussed in
several contexts.


Currently, I'm aware of the following wiki pages which each potentially
touches on some aspects of HPC:


I must admit that existing documentation/papers did not help much.
Well, did not help at all, actually.
One major problem in my opinion/experience is that
multi-core/multi-node machines can be really
different from each other, and parallelization strategies very
purpose-specific, so that creating
general-purpose documents/papers, or even software, *may* be a
hopeless effort. Smart ideas
are most welcome, of course:)


Dear Massimo and all,

Being a beginner in massively processing Landsat 8 images using JRC's
JEODPP system (which is designed for High-Throughput,
https://doi.org/10.1016/j.future.2017.11.007), I found useful notes in
the Wiki (notably Veronica's excellent tutorials) and elsewhere, got
specific answers through the mailing lists and learned a lot in on-site
discussions during the last OSGeo sprint, for example.

Nonetheless, I think to have learned quite some things the hard way. In
this regard, some answers to even "non-sense" questions are worth documenting.

My aim is to transfer notes of practical value. Having HPC and HTC
related notes in a wiki, will help to get started, promote best
practices, learn through common mistakes and give an overview for the
points Peter put in this thread's first message.

I hope it's fine to name the page "High Performance Computing". Please
advise or create a page with another name if you think otherwise.

Many thanks, Nikos


signature.asc
Description: PGP signature
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Global overview of GRASS HPC resources/installations ?

[Massi Alvioli]

2018-05-17 10:09 GMT+02:00 Moritz Lennert :

Hi,

> [I imagine your mail was supposed to go onto the mailing list and not just to 
> me...]

sure my answer was for everyone to read, I believe I tried to send it
again afterwards..
something must have gone wrong.

> I just presented GRASS and a short overview over GRASS on HPC yesterday at
> the FOSS4F-FR and there was a lot of interest for this. Several people asked
> me about specific documentation on the subject.

What we did about GRASS + HPC was for specific production purposes and
no documentation
whatsoever wascreated, basically due to lack of time.. so I find it
hard to say whether this is going
to change in the near future:). Surely the topic is of wide interest
and worth being discussed in
several contexts.

> Currently, I'm aware of the following wiki pages which each potentially
> touches on some aspects of HPC:

I must admit that existing documentation/papers did not help much.
Well, did not help at all, actually.
One major problem in my opinion/experience is that
multi-core/multi-node machines can be really
different from each other, and parallelization strategies very
purpose-specific, so that creating
general-purpose documents/papers, or even software, *may* be a
hopeless effort. Smart ideas
are most welcome, of course:)


M
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Global overview of GRASS HPC resources/installations ?

[Moritz Lennert]

On 15/05/18 17:10, Nikos Alexandris wrote:

* "Peter Löwe"  [2018-05-14 11:38:53 +0200]:


Hi all,

is there already some kind of up to date list / global map of GRASS 
installation in HPC-cluster environments ?

This would be handy to get a feeling about how much "big geospatial number 
crunching" is currently done with GRASS, and where to visit / whom to talk to to 
learn from others.

Best,
Peter


Ciao Peter, and all,

I *still* need to do my "homework" and transfer "my" notes about GRASS
GIS and HPC in the wiki. I will create a dedicated "HPC" wiki page. I
hope this will be a starting point.

As usual, all this if there are no objections.


+1

See my other mail which I wrote before seeing yours.

Moritz

___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Global overview of GRASS HPC resources/installations ?

[Moritz Lennert]

Hi Massi,

[I imagine your mail was supposed to go onto the mailing list and not 
just to me...]


On 15/05/18 14:35, Massi Alvioli wrote:

Hi GRASS HPC users,

we (Massimiliano Alvioli and Ivan Marchesini, at CNR IRPI 
http://www.irpi.cnr.it) spent a significant amonut of time using GRASS 
on local multi-core machines,
remote machines running the PBS jobs scheduler and also HPC machines, 
mainly the former Galileo and Marconi clusters and at CINECA (Italy), 
running the
SLURM job scheduler. We never published anything on the subject, but we 
will be happy to discuss with other interested people.


I just presented GRASS and a short overview over GRASS on HPC yesterday 
at the FOSS4F-FR and there was a lot of interest for this. Several 
people asked me about specific documentation on the subject.


Currently, I'm aware of the following wiki pages which each potentially 
touches on some aspects of HPC:


https://grasswiki.osgeo.org/wiki/Parallel_GRASS_jobs
https://grasswiki.osgeo.org/wiki/Working_with_GRASS_without_starting_it_explicitly
https://grasswiki.osgeo.org/wiki/GRASS_and_Shell
https://grasswiki.osgeo.org/wiki/GRASS_GIS_Performance
https://grasswiki.osgeo.org/wiki/Large_raster_data_processing
https://grasswiki.osgeo.org/wiki/Large_vector_data_processing

Then there are elements in some presentations:

https://fosdem.org/2018/schedule/event/geo_grass/
https://archive.fosdem.org/2015/schedule/event/grass_7/

This does not make it easy for people to find a one-stop place for 
guidance on how to work with GRASS in a cluster environment by 
parallelizing jobs.


So, it would be great to create one "GRASS HPC" wiki page and collect 
and structure all relevant info there.


Moritz
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Global overview of GRASS HPC resources/installations ?

[Nikos Alexandris]

* "Peter Löwe"  [2018-05-14 11:38:53 +0200]:


Hi all,

is there already some kind of up to date list / global map of GRASS 
installation in HPC-cluster environments ?

This would be handy to get a feeling about how much "big geospatial number 
crunching" is currently done with GRASS, and where to visit / whom to talk to to 
learn from others.

Best,
Peter


Ciao Peter, and all,

I *still* need to do my "homework" and transfer "my" notes about GRASS
GIS and HPC in the wiki. I will create a dedicated "HPC" wiki page. I
hope this will be a starting point.

As usual, all this if there are no objections.

Thanks, Nikos


signature.asc
Description: PGP signature
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Global overview of GRASS HPC resources/installations ?

[Markus Neteler]

On Mon, May 14, 2018 at 11:55 AM, Moritz Lennert
 wrote:
> On 14/05/18 11:38, "Peter Löwe" wrote:
>>
>> Hi all,
>>
>> is there already some kind of up to date list / global map of GRASS
>> installation in HPC-cluster environments ?
>
> I don't know of any.

Perhaps best collected in a Wiki page.

>> This would be handy to get a feeling about how much "big geospatial number
>> crunching" is currently done with GRASS, and where to visit / whom to talk
>> to to learn from others.
>
>
> +1

Yes, nice idea.

Markus
___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Global overview of GRASS HPC resources/installations ?

[Moritz Lennert]

On 14/05/18 11:38, "Peter Löwe" wrote:

Hi all,

is there already some kind of up to date list / global map of GRASS 
installation in HPC-cluster environments ?


I don't know of any.



This would be handy to get a feeling about how much "big geospatial number 
crunching" is currently done with GRASS, and where to visit / whom to talk to to 
learn from others.


+1

Moritz

___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user

[GRASS-user] Global overview of GRASS HPC resources/installations ?

[Peter Löwe]

Hi all,

is there already some kind of up to date list / global map of GRASS 
installation in HPC-cluster environments ?

This would be handy to get a feeling about how much "big geospatial number 
crunching" is currently done with GRASS, and where to visit / whom to talk to 
to learn from others.

Best,
Peter


___
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user