Hi, Andrew, Thanks for the follow up. Would you mind share a copy of your data partition with us? It seems that your data files are in a particular state that I am not able to reproduce.
In this case, you should be able to remove all your normal data files, only leave the files FBchr.??? and -part.txt in the directories. Once I get your data files, I will figure out what is the problem with the .idx files - what saw last night with my own tests was that the index for the column was not loaded into memory and therefore can not be updated when the dictionary changes in function setDictionary. This cause the function setDictionary to want to look for FBchr.int or FBchr in order to build the corresponding index.. John On 9/14/12 12:55 PM, Olson, Andrew wrote: > Hi John, > I tried 567 but it doesn't update the index because it is looking for the raw > data. You probably need to check if the column is a metaTag and set the > index accordingly. > Andrew > > thula invoking combineCategories with 1 name > Warning -- fileManager::storage::read(fname=./1/FBchr.int, begin=0, > end=35840624) failed to open the named file > Warning -- category[1.FBchr]::setDictionary failed to open data file > ./1/FBchr to create an index > Warning -- mensa[T-1]::combineCategories(FBchr) failed to update index for > 1.FBchr > > On Sep 14, 2012, at 1:37 AM, K. John Wu wrote: > >> Hi, Andrew, >> >> Looks like on the second run of the -M operation, the program >> misbehaves. This problem should be fixed in SVN revision 567. Please >> give it a try when you get the chance. >> >> Thanks. >> >> John >> >> >> >> On 9/13/12 9:05 AM, Olson, Andrew wrote: >>> Actually, -M didn't work in the case of the metaTag-derived columns, I just >>> didn't see the warnings without -v. Looking at the code, it seems there >>> needs to be a special case for metaTag-derived columns so the .idx files >>> can be regenerated without examining the underlying data (implied .int >>> file). >>> Andrew >>> >>> $ thula -d . -v -M FBchr >>> >>> Constructed a part named 1 >>> Constructed a part named 10 >>> Constructed a part named 2 >>> Constructed a part named 3 >>> Constructed a part named 4 >>> Constructed a part named 5 >>> Constructed a part named 6 >>> Constructed a part named 7 >>> Constructed a part named 8 >>> Constructed a part named 9 >>> mensa::addPartition(.) increases the number partitions from 0 to 10, the >>> number of rows from 0 to 59161937, and the number of columns from 0 to 9 >>> thula invoking combineCategories with 1 name >>> Warning -- fileManager::storage::read(fname=./1/FBchr.int, begin=0, >>> end=35840624) failed to open the named file >>> Warning -- fileManager::storage::read(fname=./10/FBchr.int, begin=0, >>> end=17331856) failed to open the named file >>> Warning -- fileManager::storage::read(fname=./2/FBchr.int, begin=0, >>> end=27313592) failed to open the named file >>> Warning -- fileManager::storage::read(fname=./3/FBchr.int, begin=0, >>> end=26749204) failed to open the named file >>> Warning -- fileManager::storage::read(fname=./4/FBchr.int, begin=0, >>> end=28259840) failed to open the named file >>> Warning -- fileManager::storage::read(fname=./5/FBchr.int, begin=0, >>> end=24325084) failed to open the named file >>> Warning -- fileManager::storage::read(fname=./6/FBchr.int, begin=0, >>> end=18849964) failed to open the named file >>> Warning -- fileManager::storage::read(fname=./7/FBchr.int, begin=0, >>> end=20123184) failed to open the named file >>> Warning -- fileManager::storage::read(fname=./8/FBchr.int, begin=0, >>> end=19779448) failed to open the named file >>> Warning -- fileManager::storage::read(fname=./9/FBchr.int, begin=0, >>> end=18074952) failed to open the named file >>> combineCategories returned 10 >>> fileManager::clear -- completed with 24 bytes of storage remain in memory >>> after removing all managed objects >>> >>> On Sep 13, 2012, at 9:37 AM, K. John Wu wrote: >>> >>>> Hi, Andrew, >>>> >>>> Great to know that the -M option works. To make sure the result table >>>> is always outputted, please use the '-x filename' option. It will >>>> write the results to the named file no matter -v is specified or not. >>>> This option should be a little faster than writing a lot of stuff to >>>> screen. >>>> >>>> This option is undocumented because it is different from the output >>>> option in other programs ;-) >>>> >>>> John >>>> >>>> >>>> On 9/13/12 5:22 AM, Olson, Andrew wrote: >>>>> Hi John, I tried thula with -M to merge the dictionaries. This >>>>> speeds up the allele_string,count(*) query to 1.65s, which is as >>>>> great! The FBchr, count(*) query doesn't return any results unless >>>>> you run with -v (but it still takes 20s that way). >>>>> >>> > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
