I changed the MAP_SHARED to MAP_PRIVATE and recompiled the code. The formatdb works fine after that small change.
-Randy > From: Sam Lang <[email protected]> > Date: Tue, 16 Jun 2009 10:08:45 -0500 > To: Randall Martin <[email protected]> > Cc: Becky Ligon <[email protected]>, Phil Carns <[email protected]>, > <[email protected]> > Subject: Re: [Pvfs2-users] help with pvfs2 and formatdb (used with BLAST) > > > Hi Randy, > > It doesn't look like they actually need to use MAP_SHARED there, > because they never modify the mapped memory region (they open the file > with O_RDONLY, and only pass PROT_READ to mmap). If they were > modifying the file elsewhere and then expecting the updates to be seen > in the mapped memory, then they would need MAP_SHARED, but it doesn't > look like they do that either. Could you change that MAP_SHARED to > MAP_PRIVATE, recompile, and run everything on PVFS again? I think we > can probably send the blast folks a patch for that, but we want to > make sure that it works first. > > Thanks, > -sam > > On Jun 16, 2009, at 8:10 AM, Randall Martin wrote: > >> We download the source from http://www.mpiblast.org/Downloads/ >> Stable. The >> mmap() call with MAP_SHARED is in mpiblast-1.5.0-pio/ncbi/corelib/ >> ncbimem.c. >> >> -Randy >> >>> From: Becky Ligon <[email protected]> >>> Reply-To: Becky Ligon <[email protected]> >>> Date: Mon, 15 Jun 2009 17:49:46 -0400 (EDT) >>> To: Sam Lang <[email protected]> >>> Cc: Becky Ligon <[email protected]>, Phil Carns >>> <[email protected]>, Randall >>> Martin <[email protected]>, <[email protected]> >>> Subject: Re: [Pvfs2-users] help with pvfs2 and formatdb (used with >>> BLAST) >>> >>> Sam: >>> >>> Randy will have to answer that question for you, since he is the >>> one who >>> actually ran the command. I'll get with him in the morning and >>> find out. >>> >>> Becky >>> >>>> >>>> Hi Becky, >>>> >>>> Does clemson build formatdb (and blast) from the source code tarball >>>> ncbi? The reason I ask is that we might be able to make a small >>>> modification (switching from MAP_SHARED to MAP_PRIVATE) to allow >>>> blast >>>> to work on PVFS deployments. I can give further info if that's >>>> something you all are willing to try. >>>> >>>> Thanks, >>>> -sam >>>> >>>> >>>> On Jun 11, 2009, at 9:18 AM, Becky Ligon wrote: >>>> >>>>> Phil: >>>>> >>>>> We have a 4Gb input file for the formatdb command. The problem >>>>> doesn't >>>>> seem to occur until after the command has crunched for several >>>>> hours. >>>>> >>>>> I'll be happy to share the file with you (and the formatdb command >>>>> that >>>>> Randy is using). Just let me know the best way to get the file to >>>>> you...I >>>>> can ftp it to you or put it on a machine that you still have access >>>>> to....just let me know! >>>>> >>>>> Becky >>>>> >>>>> >>>>>> Hi Becky, >>>>>> >>>>>> Is there an example input file (either for public download or that >>>>>> you >>>>>> could send to us) that triggers the problem? I tried this trivial >>>>>> example: >>>>>> >>>>>> http://en.wikipedia.org/wiki/Fasta_format#Header_line >>>>>> >>>>>> ... by just pasting it into a text file and running "formatdb -i >>>>>> <file>". I also tried just concatenating the same input together >>>>>> many >>>>>> times to get a bigger file, but I didn't have any luck >>>>>> triggering the >>>>>> problem. >>>>>> >>>>>> -Phil >>>>>> >>>>>> Sam Lang wrote: >>>>>>> >>>>>>> Hi Becky, >>>>>>> >>>>>>> pvfs2_file_write is a kernel module function, so I think that >>>>>>> message >>>>>>> must have come from the kernel log. There may not be any >>>>>>> messages >>>>>>> in >>>>>>> the client log (/tmp/pvfs2-client.log), but I was hoping there >>>>>>> would be >>>>>>> something that might point us at the problem (the client daemon >>>>>>> got >>>>>>> restarted, for example). Also, the server logs might have >>>>>>> something. >>>>>>> As a last resort, you could try to enable debugging in the kernel >>>>>>> module >>>>>>> to see what happens around the time of another error. >>>>>>> >>>>>>> -sam >>>>>>> >>>>>>> On Jun 9, 2009, at 3:19 PM, Becky Ligon wrote: >>>>>>> >>>>>>>> The second message (pvfs2_file_write) came out of the client log >>>>>>>> (/tmp/pvfs2-client.log). >>>>>>>> >>>>>>>> We are thinking that either the client or server processes are >>>>>>>> running >>>>>>>> out >>>>>>>> of system resources.....maybe file descriptors, for example. >>>>>>>> >>>>>>>> We have now put PVFS on our Palmetto cluster as the primary >>>>>>>> parallel >>>>>>>> file >>>>>>>> system .... no more lustre! So, now we are trying to iron >>>>>>>> some of >>>>>>>> these >>>>>>>> problems, which is why I think the problem is really system- >>>>>>>> resource >>>>>>>> related. >>>>>>>> >>>>>>>> I found an old thread from Dec of 2007 where a guy in Beijing >>>>>>>> had >>>>>>>> the >>>>>>>> same >>>>>>>> problem; unfortunately, there was no response to his problem. >>>>>>>> >>>>>>>> Becky >>>>>>>> >>>>>>>>> >>>>>>>>> Hi Becky, >>>>>>>>> What about the client log? Anything in there? >>>>>>>>> -sam >>>>>>>>> >>>>>>>>> On Jun 9, 2009, at 10:42 AM, Becky Ligon wrote: >>>>>>>>> >>>>>>>>>> PVFS-USERS: >>>>>>>>>> >>>>>>>>>> When I use formatdb to create a BLAST-usable database, I get >>>>>>>>>> the >>>>>>>>>> error: >>>>>>>>>> >>>>>>>>>> [formatdb] ERROR: Failed to create index: ISAMErrorCode -5. >>>>>>>>>> >>>>>>>>>> The PVFS kernel log shows the following error: >>>>>>>>>> >>>>>>>>>> pvfs2_file_write: error in vectored write to handle xxxx, >>>>>>>>>> FILE: >>>>>>>>>> xxxx. >>>>>>>>>> >>>>>>>>>> (the x's have values but are just specific to my database). >>>>>>>>>> >>>>>>>>>> I saw a thread in the pvfs2-users archive where someone else >>>>>>>>>> has >>>>>>>>>> this same >>>>>>>>>> problem, but there were no responses to it. Am I just >>>>>>>>>> having a >>>>>>>>>> setup >>>>>>>>>> issue with PVFS? >>>>>>>>>> >>>>>>>>>> We are running PVFS 2.8. >>>>>>>>>> >>>>>>>>>> Any help would be apprecited. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Becky Ligon >>>>>>>>>> -- >>>>>>>>>> Becky Ligon >>>>>>>>>> Clemson University >>>>>>>>>> 864-656-3865 >>>>>>>>>> _______________________________________________ >>>>>> >>>>> >>>>> >>>>> -- >>>>> Becky Ligon >>>>> PVFS Developer >>>>> Clemson University >>>>> 864-656-3865 >>>> >>>> >>> >>> >>> -- >>> Becky Ligon >>> PVFS Developer >>> Clemson University >>> 864-656-3865 >> > _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
