Hi Becky-
Excellent, thanks for the response!
chuck
On Fri, Sep 28, 2012 at 09:51:33AM -0400, Becky Ligon wrote:
> Phil Carnes believes that increasing the config file size limit is not a
> problem, as you probably have already experienced. The OFS team will add
> an item to our list of things to be done. Thanks for letting us know about
> the problem.
>
> Becky
>
> On Fri, Sep 28, 2012 at 9:22 AM, Becky Ligon <[email protected]> wrote:
>
> > Chuck:
> >
> > I think that increasing the limits are the only way, right now, for the
> > system to handle a large config file. I have a question out to Phil Carnes
> > to see if this limit change effects BMI. As soon as I hear from him, I'll
> > let you know. I believe the 65k limit was just so we could send the config
> > file in one message to the client, which is needed when a client starts up.
> > We have ways of sending larger messages but that mechanism is not
> > currently being used for the config file. BTW: PVFS can handle as many
> > servers as you want; it is just the size of the config file that is posing
> > the problem.
> >
> > We will definitely work on a better way to send a large config file, and
> > as soon as I hear from Phil, I will let you know.
> >
> > Thanks,
> > Becky
> >
> >
> > On Wed, Sep 26, 2012 at 9:49 AM, Chuck Cranor <[email protected]> wrote:
> >
> >> hi-
> >>
> >> I'm using OrangeFS 2.8.4 to help test out a large emulab-based cluster
> >> that we are putting together ( http://newmexicoconsortium.org/probe ).
> >>
> >> I am trying to bring up a 512 node PVFS cluster using pvfs2-genconfig
> >> to generate the configuration file (non-interactively, from a script).
> >> The problem I am having is that pvfs2-server starts up OK, but then dies
> >> with an assert in getconfig_init() because of
> >> PVFS_REQ_LIMIT_CONFIG_FILE_BYTES:
> >>
> >> > static PINT_sm_action getconfig_init(
> >> > struct PINT_smcb *smcb, job_status_s *js_p)
> >> > {
> >> > struct PINT_server_op *s_op = PINT_sm_frame(smcb,
> >> PINT_FRAME_CURRENT);
> >> > struct server_configuration_s *user_opts =
> >> get_server_config_struct();
> >> > assert(user_opts);
> >> >
> >> > /* TODO: handle this properly later */
> >>
> >> vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
> >> > assert(user_opts->fs_config_buflen <=
> >> PVFS_REQ_LIMIT_CONFIG_FILE_BYTES);
> >>
> >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >>
> >> I looked at the configuration file generated by pvfs2-genconfig and it
> >> is 68922 bytes. Looking at my startup log file I found:
> >>
> >> Warning: Generated config file: /etc/pvfs2-fs.conf
> >> has size: 68741, which is larger than the current PVFS request: 65536
> >> Increase the value of PVFS_REQ_LIMIT_CONFIG_FILE_BYTES in
> >> src/proto/pvfs2-req-proto.h
> >>
> >>
> >>
> >> My questions are:
> >>
> >> 1. does anyone currently use OrangeFS with 512 or more servers?
> >> If so, is the only solution to edit the source code and recompile
> >> the world, or are there other things I should consider? for
> >> example,
> >> are there ways to make smaller config files using something other
> >> than
> >> pvfs2-genconfig?
> >>
> >> 2. pvfs2-req-proto.h says if you change
> >> PVFS_REQ_LIMIT_CONFIG_FILE_BYTES
> >> you should also change $req_limit in pvfs2-genconfig as well.
> >> is that all I need to change to run on 512 nodes, or are there other
> >> things I need to change too, before recompiling?
> >>
> >> 3. if I increase the number of nodes to 1024 or larger, are there other
> >> gotchas that I should be aware of with OrangeFS? What is the
> >> biggest
> >> number of nodes you can put into an OrangeFS system?
> >>
> >>
> >>
> >> chuck
> >>
> >> _______________________________________________
> >> Pvfs2-users mailing list
> >> [email protected]
> >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
> >>
> >
> >
> >
> > --
> > Becky Ligon
> > OrangeFS Support and Development
> > Omnibond Systems
> > Anderson, South Carolina
> >
> >
> >
>
>
> --
> Becky Ligon
> OrangeFS Support and Development
> Omnibond Systems
> Anderson, South Carolina
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
