Re: [GRASS-dev] Read-access to data in a different location with same SRS
Hi all. Greetings and have nice programming sessions. (Not happy not being able to be an active member of the great team you have). Regarding linking, this https://github.com/NikosAlexandris/r.internal might be of interest. Created to not duplicate tens of thousands of Landsat maps in the High Throughput cluster (in the JRC) while generating LST time series. Cheers, Nikos --- On 2020-11-19 21:38, Helmut Kudrnovsky wrote: there and then other users including me symlink this mapset to their locations in scratch spaces, so they can read from there and work on their stuff. This has been working really well. Yes, this is a typical workflow several of us use regularly. But again, this is very easy in *nix environments, >but I'm not sure how portable it is... symlinks are available in ms windows (e.g. [1], [2], [3]), though I guess it's not widely known and used. [1] https://blogs.windows.com/windowsdeveloper/2016/12/02/symlinks-windows-10/ [2] https://www.giga.de/downloads/windows-10/tipps/symlinks-in-windows-erstellen-mit-mklink-so-gehts/ [3] https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/mklink - best regards Helmut -- Sent from: http://osgeo-org.1560.x6.nabble.com/Grass-Dev-f3991897.html ___ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev ___ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Read-access to data in a different location with same SRS
On Thu, Nov 19, 2020 at 1:16 PM Moritz Lennert wrote: > > > Am 19. November 2020 15:52:18 MEZ schrieb "Anna Petrášová" < > kratocha...@gmail.com>: > > > >On Thu, Nov 19, 2020 at 8:02 AM Moritz Lennert < > mlenn...@club.worldonline.be> > >wrote: > >> > >> But maybe we could take this idea and create a more sophisticated > >> mechanism, extending g.mapsets to allow creating a read-only access to > >> mapsets even outside the current location, as long as the reference > systems > >> are identical (would just requiring the PROJ files to be identical be > >> enough ? Too restrictive ?). > >> > > ... And I think using g.mapsets (or whatever this would then be called) to > add a subdirectory of another project to those available in the current > project would not be such a daunting challenge to new users either. > This makes a lot of sense to me by itself. To extend the search path to include mapsets in other locations (using the current terminology), g.mapsets could accept full paths, check the CRS, and store the path in the search path. The logic could be (most relevant pieces only): If the path has no backslashes, add it to the search path (mapset in current location). If the path has exactly one backslash, try if it is a location in the current database directory. If it is, check the CRS match and add it. If the above path checks failed, make it absolute path and see if it exists. If it does, check the CRS match and add it. This also falls nicely into the "simply use paths" idea (i.e., a path instead of triplet db/loc/mapset). ___ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Read-access to data in a different location with same SRS
>>there and then other users including me symlink this mapset to their >>locations in scratch spaces, so they can read from there and work on their >>stuff. This has been working really well. >> > >Yes, this is a typical workflow several of us use regularly. But again, this is very easy in *nix environments, >but I'm not sure how portable it is... symlinks are available in ms windows (e.g. [1], [2], [3]), though I guess it's not widely known and used. [1] https://blogs.windows.com/windowsdeveloper/2016/12/02/symlinks-windows-10/ [2] https://www.giga.de/downloads/windows-10/tipps/symlinks-in-windows-erstellen-mit-mklink-so-gehts/ [3] https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/mklink - best regards Helmut -- Sent from: http://osgeo-org.1560.x6.nabble.com/Grass-Dev-f3991897.html ___ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Read-access to data in a different location with same SRS
Am 19. November 2020 15:52:18 MEZ schrieb "Anna Petrášová" : >Hi, > >On Thu, Nov 19, 2020 at 8:02 AM Moritz Lennert >wrote: > >> Hi, >> >> It was great seeing some of you last night via jitsi ! >> > >indeed! > >> >> In the context of the discussion of "location" vs "project", several of us >> pointed to the fact that we organize our data into locations by reference >> system with mapsets as project directories. PERMANENT (and other mapsets) >> can then be used for data that is common to many projects and can thus be >> accessed read-only from their respective mapsets. This helps avoiding data >> duplication across projects. >> >> In the perspective where each location is a project we this would somehow >> get lost, as this concept invites duplication into each project. >> > >Right, just wanted to perhaps clarify my use case. I typically work with >high resolution data so my study areas tend to be fairly small. But they do >have common CRS (e.g. all study areas are in NC). So for me it makes sense >to create a new location (with the same CRS) for each project and then I >use mapsets to better organize my analysis, so that I don't have too many >maps in there. So the code duplication doesn't typically bother me, but I >of course I see your point. > >> >> However, Luca raised an interesting point in the chat which we didn't get >> to discuss: it might be possible to define some mechanism to make data in >> one location readable from another location as long as they are in the same >> SRS. An easy and dirty way is obviously to just use symbolic links as Luca >> seemed to suggest, but that means there is no control over the actual >> correspondence of location definitions. Also I don't know how >> cross-platform this would be. >> > >We started to use it on HPC, where we have backed-up storage with PERMANENT >location containing all the base data and only I have permissions to write >there and then other users including me symlink this mapset to their >locations in scratch spaces, so they can read from there and work on their >stuff. This has been working really well. > Yes, this is a typical workflow several of us use regularly. But again, this is very easy in *nix environments, but I'm not sure how portable it is... >> >> But maybe we could take this idea and create a more sophisticated >> mechanism, extending g.mapsets to allow creating a read-only access to >> mapsets even outside the current location, as long as the reference systems >> are identical (would just requiring the PROJ files to be identical be >> enough ? Too restrictive ?). >> >> This would possibly allow to go further in the idea of organizing data by >> projects instead of locations, while also keeping >> the idea of avoiding data duplication (people could create base data >> projects with data common to many projects). >> >> How does that sound ? >> > >Certainly worth exploring, however this seems to be more advanced, so not >sure how this would help in introducing new people to GRASS. I'm trying to find a way to reconcile your ideas behind renaming locations projects, while at the same time allowing people to continue to work in a way that resembles the original idea behind locations. Obviously, people could just continue to use projects the same way they use locations now if all we do is a simple renaming, but I have the feeling that if we really want to make step in a new direction, we should fo all the way and not just apply beautiful new gift wrap (which I'm afraid would actually cause more confusion in the long run). So my idea is to go all the way and make current locations into real autonomous projects, with mapsets relabeled as project subdirectories (or similar) thus putting the whole concept more in line with what you have been advocating for as a more user friendly approach. But making data usage across projects possible would be a prerequisite IMHO in order to allow current usage to continue. And I think using g.mapsets (or whatever this would then be called) to add a subdirectory of another project to those available in the current project would not be such a daunting challenge to new users either. But as I said yesterday, I'm not personally in demand for such change, as I don't find the current location and mapset concept so difficult to convey, so I won't insist on continuing along these lines of thought . ;-) Moritz ___ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Read-access to data in a different location with same SRS
Hi, On Thu, Nov 19, 2020 at 8:02 AM Moritz Lennert wrote: > Hi, > > It was great seeing some of you last night via jitsi ! > indeed! > > In the context of the discussion of "location" vs "project", several of us > pointed to the fact that we organize our data into locations by reference > system with mapsets as project directories. PERMANENT (and other mapsets) > can then be used for data that is common to many projects and can thus be > accessed read-only from their respective mapsets. This helps avoiding data > duplication across projects. > > In the perspective where each location is a project we this would somehow > get lost, as this concept invites duplication into each project. > Right, just wanted to perhaps clarify my use case. I typically work with high resolution data so my study areas tend to be fairly small. But they do have common CRS (e.g. all study areas are in NC). So for me it makes sense to create a new location (with the same CRS) for each project and then I use mapsets to better organize my analysis, so that I don't have too many maps in there. So the code duplication doesn't typically bother me, but I of course I see your point. > > However, Luca raised an interesting point in the chat which we didn't get > to discuss: it might be possible to define some mechanism to make data in > one location readable from another location as long as they are in the same > SRS. An easy and dirty way is obviously to just use symbolic links as Luca > seemed to suggest, but that means there is no control over the actual > correspondence of location definitions. Also I don't know how > cross-platform this would be. > We started to use it on HPC, where we have backed-up storage with PERMANENT location containing all the base data and only I have permissions to write there and then other users including me symlink this mapset to their locations in scratch spaces, so they can read from there and work on their stuff. This has been working really well. > > But maybe we could take this idea and create a more sophisticated > mechanism, extending g.mapsets to allow creating a read-only access to > mapsets even outside the current location, as long as the reference systems > are identical (would just requiring the PROJ files to be identical be > enough ? Too restrictive ?). > > This would possibly allow to go further in the idea of organizing data by > projects instead of locations, while also keeping > the idea of avoiding data duplication (people could create base data > projects with data common to many projects). > > How does that sound ? > Certainly worth exploring, however this seems to be more advanced, so not sure how this would help in introducing new people to GRASS. Anna > > Moritz > ___ > grass-dev mailing list > grass-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/grass-dev ___ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev
Re: [GRASS-dev] Read-access to data in a different location with same SRS
On Thu, 19 Nov 2020 at 14:02, Moritz Lennert wrote: > > Hi, > Hi, > It was great seeing some of you last night via jitsi ! > > In the context of the discussion of "location" vs "project", several of us > pointed to the fact that we organize our data into locations by reference > system with mapsets as project directories. PERMANENT (and other mapsets) can > then be used for data that is common to many projects and can thus be > accessed read-only from their respective mapsets. This helps avoiding data > duplication across projects. > > In the perspective where each location is a project we this would somehow get > lost, as this concept invites duplication into each project. > > However, Luca raised an interesting point in the chat which we didn't get to > discuss: it might be possible to define some mechanism to make data in one > location readable from another location as long as they are in the same SRS. > An easy and dirty way is obviously to just use symbolic links as Luca seemed > to suggest, but that means there is no control over the actual correspondence > of location definitions. Also I don't know how cross-platform this would be. > > But maybe we could take this idea and create a more sophisticated mechanism, > extending g.mapsets to allow creating a read-only access to mapsets even > outside the current location, as long as the reference systems are identical > (would just requiring the PROJ files to be identical be enough ? Too > restrictive ?). > > This would possibly allow to go further in the idea of organizing data by > projects instead of locations, while also keeping > the idea of avoiding data duplication (people could create base data projects > with data common to many projects). > > How does that sound ? > I support any idea that could be useful for GRASS, and I think project could bring more users, however I would like to continue to work as I do today and I think this is the main idea of Vaclav, if I understood correctly. If we want switch to project I think we should remove all the references to Location (in doc and code) > Moritz > -- ciao Luca www.lucadelu.org ___ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev
[GRASS-dev] Read-access to data in a different location with same SRS
Hi, It was great seeing some of you last night via jitsi ! In the context of the discussion of "location" vs "project", several of us pointed to the fact that we organize our data into locations by reference system with mapsets as project directories. PERMANENT (and other mapsets) can then be used for data that is common to many projects and can thus be accessed read-only from their respective mapsets. This helps avoiding data duplication across projects. In the perspective where each location is a project we this would somehow get lost, as this concept invites duplication into each project. However, Luca raised an interesting point in the chat which we didn't get to discuss: it might be possible to define some mechanism to make data in one location readable from another location as long as they are in the same SRS. An easy and dirty way is obviously to just use symbolic links as Luca seemed to suggest, but that means there is no control over the actual correspondence of location definitions. Also I don't know how cross-platform this would be. But maybe we could take this idea and create a more sophisticated mechanism, extending g.mapsets to allow creating a read-only access to mapsets even outside the current location, as long as the reference systems are identical (would just requiring the PROJ files to be identical be enough ? Too restrictive ?). This would possibly allow to go further in the idea of organizing data by projects instead of locations, while also keeping the idea of avoiding data duplication (people could create base data projects with data common to many projects). How does that sound ? Moritz ___ grass-dev mailing list grass-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/grass-dev
