Re: [GRASS-user] Splitting a location across several disks

[Blumentrath, Stefan]

See also: https://grasswiki.osgeo.org/wiki/Location_and_Mapsets
There you find a detailed description of how to link mapsets...

-Original Message-
From: grass-user-boun...@lists.osgeo.org 
[mailto:grass-user-boun...@lists.osgeo.org] On Behalf Of Dylan Beaudette
Sent: 8. oktober 2015 00:50
To: Glynn Clements <gl...@gclements.plus.com>
Cc: grass list <grass-user@lists.osgeo.org>
Subject: Re: [GRASS-user] Splitting a location across several disks

On Wed, Oct 7, 2015 at 3:39 PM, Glynn Clements <gl...@gclements.plus.com> wrote:
>
>
> Dylan Beaudette wrote:
>
> > It has been a while, but glad to be back on GRASS-user.
> >
> > I am working on a project that involves a significant storage 
> > dilemma: try and fit most of the files into a 500 Gb SSD for 
> > blazing-fast I/O, or fall back to a standard but higher capacity disk drive.
> >
> > Would it be possible to store "derived" data into a mapset that is 
> > on standard disk, while the "source" data reside in another mapset, 
> > stored on the SSD?
> >
> > In other words, is it OK for a location to contain several mapsets 
> > that don't "live" on the same physical disk. It seems like it should 
> > work (via symlink), but I would like to see if there are any caveats 
> > that I should be aware of.
>
> If symlinks don't work, Linux supports "mount --bind ...", which lets 
> you mount a directory from an already-mounted filesystem at another 
> location. Windows has similar features (e.g. reparse points), although 
> I'm not that familiar with the specifics.
>

Excellent! This is the answer that I was looking for. I will try symlinks 
first, otherwise the "mount --bind" strategy is simple enough. Fortunately this 
work will be done on linux so just about anything is possible.

>
> The main constraint is that you can't split a single mapset across 
> devices, as it must be possible to rename() files in the .tmp 
> subdirectory to other directories withing the mapset, which requires 
> that they are on the same physical partition (rename() only 
> manipulates directory entries, it won't move the file's data blocks).

OK, good to know.

Is there any reason to think that reading lots of raster files will be 
noticeably faster on the SSD?

Thanks,
Dylan

> --
> Glynn Clements <gl...@gclements.plus.com>
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Splitting a location across several disks

[Dylan Beaudette]

Thanks, this is quite helpful. I'll report back with my strategy and
any limitations of this approach.

On Wed, Oct 7, 2015 at 11:43 PM, Blumentrath, Stefan
<stefan.blumentr...@nina.no> wrote:
> See also: https://grasswiki.osgeo.org/wiki/Location_and_Mapsets
> There you find a detailed description of how to link mapsets...
>
> -Original Message-
> From: grass-user-boun...@lists.osgeo.org 
> [mailto:grass-user-boun...@lists.osgeo.org] On Behalf Of Dylan Beaudette
> Sent: 8. oktober 2015 00:50
> To: Glynn Clements <gl...@gclements.plus.com>
> Cc: grass list <grass-user@lists.osgeo.org>
> Subject: Re: [GRASS-user] Splitting a location across several disks
>
> On Wed, Oct 7, 2015 at 3:39 PM, Glynn Clements <gl...@gclements.plus.com> 
> wrote:
>>
>>
>> Dylan Beaudette wrote:
>>
>> > It has been a while, but glad to be back on GRASS-user.
>> >
>> > I am working on a project that involves a significant storage
>> > dilemma: try and fit most of the files into a 500 Gb SSD for
>> > blazing-fast I/O, or fall back to a standard but higher capacity disk 
>> > drive.
>> >
>> > Would it be possible to store "derived" data into a mapset that is
>> > on standard disk, while the "source" data reside in another mapset,
>> > stored on the SSD?
>> >
>> > In other words, is it OK for a location to contain several mapsets
>> > that don't "live" on the same physical disk. It seems like it should
>> > work (via symlink), but I would like to see if there are any caveats
>> > that I should be aware of.
>>
>> If symlinks don't work, Linux supports "mount --bind ...", which lets
>> you mount a directory from an already-mounted filesystem at another
>> location. Windows has similar features (e.g. reparse points), although
>> I'm not that familiar with the specifics.
>>
>
> Excellent! This is the answer that I was looking for. I will try symlinks 
> first, otherwise the "mount --bind" strategy is simple enough. Fortunately 
> this work will be done on linux so just about anything is possible.
>
>>
>> The main constraint is that you can't split a single mapset across
>> devices, as it must be possible to rename() files in the .tmp
>> subdirectory to other directories withing the mapset, which requires
>> that they are on the same physical partition (rename() only
>> manipulates directory entries, it won't move the file's data blocks).
>
> OK, good to know.
>
> Is there any reason to think that reading lots of raster files will be 
> noticeably faster on the SSD?
>
> Thanks,
> Dylan
>
>> --
>> Glynn Clements <gl...@gclements.plus.com>
> ___
> grass-user mailing list
> grass-user@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-user
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Splitting a location across several disks

[Dylan Beaudette]

On Wed, Oct 7, 2015 at 3:39 PM, Glynn Clements  wrote:
>
>
> Dylan Beaudette wrote:
>
> > It has been a while, but glad to be back on GRASS-user.
> >
> > I am working on a project that involves a significant storage dilemma: try
> > and fit most of the files into a 500 Gb SSD for blazing-fast I/O, or fall
> > back to a standard but higher capacity disk drive.
> >
> > Would it be possible to store "derived" data into a mapset that is on
> > standard disk, while the "source" data reside in another mapset, stored on
> > the SSD?
> >
> > In other words, is it OK for a location to contain several mapsets that
> > don't "live" on the same physical disk. It seems like it should work (via
> > symlink), but I would like to see if there are any caveats that I should be
> > aware of.
>
> If symlinks don't work, Linux supports "mount --bind ...", which lets
> you mount a directory from an already-mounted filesystem at another
> location. Windows has similar features (e.g. reparse points), although
> I'm not that familiar with the specifics.
>

Excellent! This is the answer that I was looking for. I will try
symlinks first, otherwise the "mount --bind" strategy is simple
enough. Fortunately this work will be done on linux so just about
anything is possible.

>
> The main constraint is that you can't split a single mapset across
> devices, as it must be possible to rename() files in the .tmp
> subdirectory to other directories withing the mapset, which requires
> that they are on the same physical partition (rename() only
> manipulates directory entries, it won't move the file's data blocks).

OK, good to know.

Is there any reason to think that reading lots of raster files will be
noticeably faster on the SSD?

Thanks,
Dylan

> --
> Glynn Clements 
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

[GRASS-user] Splitting a location across several disks

[Dylan Beaudette]

Hi,

It has been a while, but glad to be back on GRASS-user.

I am working on a project that involves a significant storage dilemma: try
and fit most of the files into a 500 Gb SSD for blazing-fast I/O, or fall
back to a standard but higher capacity disk drive.

Would it be possible to store "derived" data into a mapset that is on
standard disk, while the "source" data reside in another mapset, stored on
the SSD?

In other words, is it OK for a location to contain several mapsets that
don't "live" on the same physical disk. It seems like it should work (via
symlink), but I would like to see if there are any caveats that I should be
aware of.

Also, is the NULL file compression code ready for production use?

https://trac.osgeo.org/grass/ticket/2750


Thanks!
Dylan
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user

Re: [GRASS-user] Splitting a location across several disks

[Glynn Clements]

Dylan Beaudette wrote:

> It has been a while, but glad to be back on GRASS-user.
> 
> I am working on a project that involves a significant storage dilemma: try
> and fit most of the files into a 500 Gb SSD for blazing-fast I/O, or fall
> back to a standard but higher capacity disk drive.
> 
> Would it be possible to store "derived" data into a mapset that is on
> standard disk, while the "source" data reside in another mapset, stored on
> the SSD?
> 
> In other words, is it OK for a location to contain several mapsets that
> don't "live" on the same physical disk. It seems like it should work (via
> symlink), but I would like to see if there are any caveats that I should be
> aware of.

If symlinks don't work, Linux supports "mount --bind ...", which lets
you mount a directory from an already-mounted filesystem at another
location. Windows has similar features (e.g. reparse points), although
I'm not that familiar with the specifics.

The main constraint is that you can't split a single mapset across
devices, as it must be possible to rename() files in the .tmp
subdirectory to other directories withing the mapset, which requires
that they are on the same physical partition (rename() only
manipulates directory entries, it won't move the file's data blocks).

-- 
Glynn Clements 
___
grass-user mailing list
grass-user@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-user