Re: [gpfsug-discuss] GPFS FPO

[Aaron Knister]

To tell you the truth, I don't. It's on my radar but I haven't done it 
yet. I *have* run hadoop on GPFS w/o magpie though and on only a couple 
of nodes was able to pound 1GB/s out to GPFS w/ the terasort benchmark. 
I know our GPFS FS can go much faster than that but java was cpu-bound 
as it often seems to be.


-Aaron

On 8/23/16 7:56 AM, Brian Marshall wrote:

Aaron,

Do you have experience running this on native GPFS?  The docs say Lustre
and any NFS filesystem.

Thanks,
Brian


On Aug 22, 2016 10:37 PM, "Aaron Knister" > wrote:

Yes, indeed. Note that these are my personal opinions.

It seems to work quite well and it's not terribly hard to set up or
get running. That said, if you've got a traditional HPC cluster with
reasonably good bandwidth (and especially if your data is already on
the HPC cluster) I wouldn't bother with FPO and just use something
like magpie (https://github.com/LLNL/magpie
) to run your hadoopy workload on
GPFS on your traditional HPC cluster. I believe FPO (and by
extension data locality) is important when the available bandwidth
between your clients and servers/disks (in a traditional GPFS
environment) is less than the bandwidth available within a node
(e.g. between your local disks and the host CPU).

-Aaron

On 8/22/16 10:23 PM, Brian Marshall wrote:

Does anyone have any experiences to share (good or bad) about
setting up
and utilizing FPO for hadoop compute on top of GPFS?


___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org 
http://gpfsug.org/mailman/listinfo/gpfsug-discuss



--
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776 
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org 
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss



--
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

Re: [gpfsug-discuss] GPFS FPO

[Brian Marshall]

Aaron,

Do you have experience running this on native GPFS?  The docs say Lustre
and any NFS filesystem.

Thanks,
Brian

On Aug 22, 2016 10:37 PM, "Aaron Knister"  wrote:

> Yes, indeed. Note that these are my personal opinions.
>
> It seems to work quite well and it's not terribly hard to set up or get
> running. That said, if you've got a traditional HPC cluster with reasonably
> good bandwidth (and especially if your data is already on the HPC cluster)
> I wouldn't bother with FPO and just use something like magpie (
> https://github.com/LLNL/magpie) to run your hadoopy workload on GPFS on
> your traditional HPC cluster. I believe FPO (and by extension data
> locality) is important when the available bandwidth between your clients
> and servers/disks (in a traditional GPFS environment) is less than the
> bandwidth available within a node (e.g. between your local disks and the
> host CPU).
>
> -Aaron
>
> On 8/22/16 10:23 PM, Brian Marshall wrote:
>
>> Does anyone have any experiences to share (good or bad) about setting up
>> and utilizing FPO for hadoop compute on top of GPFS?
>>
>>
>> ___
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>>
> --
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776
> ___
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
___
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss