Hi, Jilong, The surprising information from the log file is that the evaluate of condition "COL_B=0" is somehow taking much longer with the new version. Would you mind rerun the ibis command line with -q "where COLB=0" and -v 8? I was suspecting the problem was related to dictionary, but it appears to not..
Thanks John On 6/4/15 4:45 PM, wang jilong wrote: > Hi John, > > Thanks. Attached please find two log files with verbose level=4, with version > 2.0.2 and 1.3.8, > using "ibis" included in the source code tree. (I have removed/masked > sensitive info.) > > Duration in 1.3.8 version is about 0.104sec; > Duration in 2.0.2 version is about 0.332sec; > > In my observation in our real code, the performance downgrade may be also in > other areas, > as the we are running similar queries on same columns, all of them are slower. > > FYI: the data/index files were built using 1.3.8. > > Thanks > > Jilong Wang > > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of K. John Wu > Sent: Thursday, June 04, 2015 1:02 PM > To: FastBit Users > Subject: Re: [FastBit-users] FastBit 2.0.2 is 3X slower on index files built > using 1.3.8? > > Hi, Jilong, > > This is might have something to how the dictionary used for keyword > indexes are initialized. Since you are writing a program calling > FastBit, presumably, you might be processing many queries in your > program. In which case, this increased start up time might not matter > too much. However, to know this for sure, would you mind turn up > verboseness level to 4 or above and send us the log file. This will > allow us to know for sure where the time is spent in your program. > > By the way, in your program, you can increase the verboseness level by > calling: > > ibis::util::setVerboseLevel(4); > > > Good luck. > > John > > > On 5/22/15 4:45 PM, wang jilong wrote: >> Hi John, >> >> We have compared two cases: >> Case 1. running programs built with fastbit 1.3.8, on index files >> built using 1.3.8. >> Case 2. running programs built with fastbit 2.0.2, on index files >> built using 1.3.8. >> >> With everything same, and no single line change on our programs. The case 1 >> was run first. >> The case 1 (1.3.8 program on 1.3.8 index files) is at least 3X faster. >> >> The comparison using "2.0.2 ibis" and "1.3.8 ibis" showed the same behavior. >> >> Out query is simple: 'select count(1) WHERE COL_A contains "com.google.com" >> and COL_B=0'. >> >> There are about 60M records in index files. >> >> COL_A, COL_B are defined below as in "-part.txt": >> >> ===================================================================== >> Begin Column >> name = "COL_A" >> description = COL_A >> data_type = "TEXT" >> index=keywords delimiters=";" >> End Column >> >> Begin Column >> name = "COL_B" >> data_type = "INT" >> minimum = 0 >> maximum = 2 >> index = <binning none/><encoding equality/> >> End Column >> ===================================================================== >> >> Question: Is it the expected behavior, or we did something wrong? >> >> Thanks >> >> Jilong Wang >> >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
