Resending! Becky
On Mon, Oct 27, 2014 at 4:02 PM, Becky Ligon <[email protected]> wrote: > Steve: > > The problem with your original config file is that all the servers in one > particular config file apply to all of the filesystems defined in that > config file. I have attached two config files that separate the > filesystems. > > To create the storage space: > > pvfs2-server -f local.config -a local > pvfs2-server -f shared.config -a shared > > To start the servers: > pvfs2-server local.config -a local > pvfs2-server shared.config -a shared > > pvfs2tab file: > tcp://localhost:3334/local /mnt/local pvfs2 default,noauto 0 0 > tcp://localhost:3335/shared /mnt/shared pvfs2 default,noauto 0 0 > > > > > > TIPS: > 1. Do not run an OrangeFS filesystem on top of NFS. It works but caching > provided by NFS will sometimes cause OrangeFS to return faulty > information. OrangeFS should have direct access to the underlying storage > layer. > > 2. Do not run executables directly from OrangeFS. It works but is slow > and therefore not a good practice. Have your users copy executables from > OrangeFS to the local machine and execute from there. > > 3. Do not compile on OrangeFS. Again, it works but is slow. The system > is not designed for the short transfers needed for a compile(as is true for > running executables). > > 4. For MPI-IO (pHDF5), a standard install of OrangeFS generally works the > best. Having multiple server machines running one OrangeFS server per > machine is the ideal setup. OrangeFS was originally developed to support > MPI-IO. If you can give me more info about your storage, I can provide > additional recommendations. > > Hope this helps! > > Becky > > On Mon, Oct 27, 2014 at 11:35 AM, Steven Varga <[email protected]> > wrote: > >> Becky: >> >> Thank you very much! Will look at it as soon as I can and confirm if >> worked. >> >> Steve >> On Oct 27, 2014 11:06 AM, "Becky Ligon" <[email protected]> wrote: >> >>> Steve: >>> >>> I will send you a config file that will do what you want but I will also >>> send some tips on other options as well. >>> >>> Becky >>> >>> Sent from my iPhone >>> >>> On Oct 27, 2014, at 10:37 AM, Steven Varga <[email protected]> >>> wrote: >>> >>> Becky: >>> >>> I tried several linear combinations of the setup I sent and it >>> complained about the handle ranges or didn't recognize the second server >>> during the creation of databases. The range values are non overlapping >>> sequential blocks. >>> >>> I only can send you the output tomorrow, as I am away. >>> >>> I browsed through the source to see how parsing is done - I couldn't >>> find a quick solution. I am not certain if separate server options needed >>> or they should be lumped together? Is it my interpretation wrong with >>> setting up multiple servers on a node or there is something else to it. >>> >>> The single server works well both with mpi calls and kernel mudule. >>> pinging is OK and copy throughput is exceptionally good. >>> >>> Steve >>> On Oct 27, 2014 10:10 AM, "Becky Ligon" <[email protected]> wrote: >>> >>>> Steve: >>>> >>>> What kind of errors are you seeing? Do the servers start at all? >>>> >>>> Becky >>>> >>>> On Sun, Oct 26, 2014 at 9:04 PM, Steven Varga <[email protected]> >>>> wrote: >>>> >>>>> Hello Becky, >>>>> >>>>> thank you for your quick response. The problem I would like to solve >>>>> is to eliminate NFS from cluster setup which serves common files across >>>>> nodes. This is the shared context. >>>>> Whereas providing file locality to MPI-IO context can significantly >>>>> reduce network throughout. This is the local context. >>>>> >>>>> The shared context is used to distribute executables across nodes, and >>>>> local context is for large pHDF5 files accessed through MPIO interface in >>>>> parallel. >>>>> >>>>> From the documentation I assembled a configuration file which doesn't >>>>> work, >>>>> steve >>>>> >>>>> >>>>> ---------------- /etc/pvfs2-fs.conf ------------------------------ >>>>> >>>>> <Defaults> >>>>> UnexpectedRequests 50 >>>>> EventLogging none >>>>> EnableTracing no >>>>> LogStamp datetime >>>>> BMIModules bmi_tcp >>>>> FlowModules flowproto_multiqueue >>>>> PerfUpdateInterval 1000 >>>>> ServerJobBMITimeoutSecs 30 >>>>> ServerJobFlowTimeoutSecs 30 >>>>> ClientJobBMITimeoutSecs 300 >>>>> ClientJobFlowTimeoutSecs 300 >>>>> ClientRetryLimit 5 >>>>> ClientRetryDelayMilliSecs 2000 >>>>> PrecreateBatchSize 0,32,512,32,32,32,0 >>>>> PrecreateLowThreshold 0,16,256,16,16,16,0 >>>>> >>>>> LogFile /var/log/orangefs-server.log >>>>> </Defaults> >>>>> >>>>> <Aliases> >>>>> Alias shared tcp://localhost:3334 >>>>> Alias local tcp://localhost:3335 >>>>> </Aliases> >>>>> >>>>> <ServerOptions> >>>>> Server local >>>>> DataStorageSpace /usr/local/storage/local/data >>>>> MetadataStorageSpace /usr/local/storage/local/meta >>>>> </ServerOptions> >>>>> >>>>> <ServerOptions> >>>>> Server shared >>>>> DataStorageSpace /usr/local/storage/shared/data >>>>> MetadataStorageSpace /usr/local/storage/shared/meta >>>>> </ServerOptions> >>>>> >>>>> >>>>> <Filesystem> >>>>> Name shared >>>>> ID 100011 >>>>> RootHandle 100021 >>>>> FileStuffing yes >>>>> <MetaHandleRanges> >>>>> Range local 100001-200000 >>>>> </MetaHandleRanges> >>>>> >>>>> <DataHandleRanges> >>>>> Range local 400001-500000 >>>>> </DataHandleRanges> >>>>> <StorageHints> >>>>> TroveSyncMeta yes >>>>> TroveSyncData no >>>>> TroveMethod alt-aio >>>>> </StorageHints> >>>>> </Filesystem> >>>>> >>>>> <Filesystem> >>>>> Name local >>>>> ID 200011 >>>>> RootHandle 200021 >>>>> FileStuffing yes >>>>> <MetaHandleRanges> >>>>> Range local 200001-400000 >>>>> </MetaHandleRanges> >>>>> >>>>> <DataHandleRanges> >>>>> Range local 500001-600000 >>>>> </DataHandleRanges> >>>>> <StorageHints> >>>>> TroveSyncMeta yes >>>>> TroveSyncData no >>>>> TroveMethod alt-aio >>>>> </StorageHints> >>>>> </Filesystem> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Sun, Oct 26, 2014 at 8:49 PM, Becky Ligon <[email protected]> >>>>> wrote: >>>>> >>>>>> Steve: >>>>>> >>>>>> I can help you setup your system but I need more specific >>>>>> information. What do you mean, exactly, by a shared storage space and a >>>>>> local storage space? >>>>>> >>>>>> Becky >>>>>> >>>>>> On Sun, Oct 26, 2014 at 5:22 PM, Steven Varga <[email protected] >>>>>> > wrote: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I am trying to configure OrangeFS with two different storage place; >>>>>>> one shared among nodes while the other used only locally; >>>>>>> >>>>>>> While the documentation suggest that it is possible to have multiple >>>>>>> server sections and servers per node, there is no example to show the >>>>>>> right >>>>>>> way of doing it. >>>>>>> >>>>>>> Did anyone try similar setup with success? >>>>>>> >>>>>>> thanks, >>>>>>> steve >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Pvfs2-users mailing list >>>>>>> [email protected] >>>>>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >>>>>>> >>>>>>> >>>>>> >>>>> >>>> > > > -- > Becky Ligon > Research Associate > Clemson University > Clemson, SC >
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
