Hi, Alexandre, My apologies. I have messed up the modification you've given me earlier.
We are very appreciative of your test data and test case. The problem should be fixed now -- there was an extra ++j statement in my version of the code. The change has been rolled into the new release ibis1.1.4. You may find it at <https://codeforge.lbl.gov/frs/download.php/124/fastbit-ibis1.1.4.tar.gz>. A description of the changes is available at <https://codeforge.lbl.gov/frs/shownotes.php?release_id=123>. Please give this new code a try and let us know if you find any issues. Thanks. John On 11/10/2009 7:41 AM, Alexandre Maurel wrote: > Hi John, > > I update my code with fastbit snapshot fastbit-scm-2009-09-01. > I did some test and it is not working for me. > > To make it work, I had to go back to the implementation of the segment > method I first sent. > > As attached document you will find the dataset I used for the test. > > The dataset contains 152 rows, when I run a query (select <all columns>) > that should return the complete dataset, 2 rows are missing. > > Here are the segment methods implementation I used to make it work. > > array_t<uint32_t>* > ibis::colFloats::segment(const array_t<uint32_t>* old) const { > array_t<uint32_t>* res = new array_t<uint32_t>; > // res->push_back(0); // the first number is always 0 > uint32_t j = 0; > float target = *(array->begin()); > const uint32_t nelm = array->size(); > > if (old != 0 && old->size()>2) { > // find segments with in the previously defined segments > for (uint32_t i=0; i<old->size()-1; ++i) { > if (j < nelm) { > res->push_back(j); > target = (*array)[j]; > ++ j; > if (j < nelm) { > do { > while (j < (*old)[i+1] && (*array)[j] == > target) ++ > j; > if (j < (*old)[i+1]) { > res->push_back(j); > if (j < nelm) { > target = (*array)[j]; > ++ j; > } > } > } while (j < (*old)[i+1]); > } > }; > } > } > else { // start with all elements in one segment > res->push_back(0); // the first number is always 0 > uint32_t j = 1; > while (j < nelm) { > while (j < nelm && (*array)[j] == target) > ++ j; > res->push_back(j); > if (j < nelm) { > target = (*array)[j]; > ++ j; > } > } > } > if (res->back() < nelm) > res->push_back(nelm); > return res; > } // ibis::colFloats::segment > > and > > ibis::array_t<uint32_t>* > ibis::colUInts::segment(const array_t<uint32_t>* old) const { > array_t<uint32_t>* res = new array_t<uint32_t>; > // res->push_back(0); // the first number is always 0 > uint32_t j = 0; > uint32_t target = *(array->begin()); > const uint32_t nelm = array->size(); > > if (old != 0 && old->size()>2) { > // find segments with in the previously defined segments > for (uint32_t i=0; i<old->size()-1; ++i) { > if (j < nelm) { > res->push_back(j); > target = (*array)[j]; > ++ j; > if (j < nelm) { > do { > while (j < (*old)[i+1] && (*array)[j] == target) ++ > j; > if (j < (*old)[i+1]) { > res->push_back(j); > if (j < nelm) { > target = (*array)[j]; > ++ j; > } > } > } while (j < (*old)[i+1]); > } > }; > } > } > else { // start with all elements in one segment > res->push_back(0); // the first number is always 0 > uint32_t j = 1; > while (j < nelm) { > while (j < nelm && (*array)[j] == target) > ++ j; > res->push_back(j); > if (j < nelm) { > target = (*array)[j]; > ++ j; > } > } > } > if (res->back() < nelm) > res->push_back(nelm); > return res; > } // ibis::colUInts::segment > > Alexandre > > [email protected] a écrit : >> Send FastBit-users mailing list submissions to >> [email protected] >> >> To subscribe or unsubscribe via the World Wide Web, visit >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> or, via email, send a message with subject or body 'help' to >> [email protected] >> >> You can reach the person managing the list at >> [email protected] >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of FastBit-users digest..." >> >> >> Today's Topics: >> >> 1. Re: problem in segment method (K. John Wu) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Fri, 28 Aug 2009 06:56:19 -0700 >> From: "K. John Wu" <[email protected]> >> Subject: Re: [FastBit-users] problem in segment method >> To: FastBit Users <[email protected]> >> Message-ID: <[email protected]> >> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >> >> Hi, Alexandre, >> >> You are right, I misunderstood your original message and only realized >> my mistake after I have sent out the first message. We have adopted >> your changes to ibis::colFloats::segment and have made changes to >> other versions of the function segment. >> >> Hope you have got a chance to try the latest snapshot >> <https://codeforge.lbl.gov/snapshots.php?group_id=44>. Please let us >> know if it correctly resolves the problem for you. >> >> Again, thanks for the feedback and your interest in FastBit. >> >> John >> >> PS: Since we have accepted your modified version of >> ibis::colFloats::segment, your name is now listed in the list of >> contributors >> <https://codeforge.lbl.gov/plugins/scmsvn/viewcvs.php/trunk/AUTHORS?root=fastbit&view=markup> >> >> along with a more detailed information in ChangeLog >> <https://codeforge.lbl.gov/plugins/scmsvn/viewcvs.php/trunk/ChangeLog?root=fastbit&view=markup>. >> >> >> >> On 8/28/2009 2:13 AM, Alexandre Maurel wrote: >> >>> Hello John, >>> >>> You suspect that the last row is not in the table because of a >>> missing end-of-line, but it is not the case. >>> If you look at the last column of the result set (with counts), the >>> last row has counts equals to 2. >>> >>> Moreover the following query : >>> ibis -d tmp -q "select Barcode,Row,Col,Cycle,Sample where Row = 4" >>> return the correct result. >>> >>> Barcode, Row, Col, Cycle, Sample (with counts) >>> 100220, 4, 4, 38, sample2, 1 >>> >>> which is the last row of the table. >>> >>> Note that the segment method implementation I sent is working >>> properly on this data set. >>> >>> Alexandre >>> >>> 2009/8/27 <[email protected] >>> <mailto:[email protected]>> >>> >>> Send FastBit-users mailing list submissions to >>> [email protected] >>> <mailto:[email protected]> >>> >>> To subscribe or unsubscribe via the World Wide Web, visit >>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>> or, via email, send a message with subject or body 'help' to >>> [email protected] >>> <mailto:[email protected]> >>> >>> You can reach the person managing the list at >>> [email protected] >>> <mailto:[email protected]> >>> >>> When replying, please edit your Subject line so it is more specific >>> than "Re: Contents of FastBit-users digest..." >>> >>> >>> Today's Topics: >>> >>> 1. problem in segment method (Alexandre Maurel) >>> 2. problem in segment method (Alexandre Maurel) >>> 3. Re: problem in segment method (K. John Wu) >>> >>> >>> >>> ---------------------------------------------------------------------- >>> >>> Message: 1 >>> Date: Thu, 27 Aug 2009 17:55:57 +0200 >>> From: Alexandre Maurel <[email protected] >>> <mailto:[email protected]>> >>> Subject: [FastBit-users] problem in segment method >>> To: [email protected] >>> <mailto:[email protected]> >>> Message-ID: >>> <[email protected] >>> >>> <mailto:[email protected]>> >>> Content-Type: text/plain; charset="iso-8859-1" >>> >>> Hello, >>> >>> I used ardea to generate a table using the following : >>> >>> Barcode Row Col Cycle Sample >>> 100220 1 1 30 sample1 >>> 100220 1 1 36 sample1 >>> 100220 1 2 30 sample1 >>> 100220 1 2 36 sample1 >>> 100220 2 1 30 sample1 >>> 100220 2 1 36 sample1 >>> 100220 2 2 30 sample1 >>> 100220 2 2 36 sample1 >>> 100220 3 3 38 sample2 >>> 100220 4 4 38 sample2 >>> >>> ardea -d tmp -m >>> "Barcode:float,Row:float,Col:float,Cycle:float,Sample:category" -t >>> Array.txt >>> >>> then using ibis, >>> >>> ibis -d tmp -q "select Barcode,Row,Col,Cycle,Sample where Row > 0" >>> >>> result gives .... >>> >>> Barcode, Row, Col, Cycle, Sample (with counts) >>> 100220, 1, 1, 30, sample1, 1 >>> 100220, 1, 1, 36, sample1, 1 >>> 100220, 1, 2, 30, sample1, 1 >>> 100220, 1, 2, 36, sample1, 1 >>> 100220, 2, 1, 30, sample1, 1 >>> 100220, 2, 1, 36, sample1, 1 >>> 100220, 2, 2, 30, sample1, 1 >>> 100220, 2, 2, 36, sample1, 1 >>> 100220, 3, 3, 38, sample2, 2 >>> >>> which is wrong the last row is missing .... >>> >>> Here how I solved this bug >>> >>> array_t<uint32_t>* >>> ibis::colFloats::segment(const array_t<uint32_t>* old) const { >>> array_t<uint32_t>* res = new array_t<uint32_t>; >>> // res->push_back(0); // the first number is always 0 >>> uint32_t j = 0; >>> float target = *(array->begin()); >>> const uint32_t nelm = array->size(); >>> >>> if (old != 0 && old->size()>2) { >>> // find segments with in the previously defined segments >>> for (uint32_t i=0; i<old->size()-1; ++i) { >>> if (j < nelm) { >>> res->push_back(j); >>> target = (*array)[j]; >>> ++ j; >>> if (j < nelm) { >>> do { >>> while (j < (*old)[i+1] && (*array)[j] == >>> target) ++ >>> j; >>> if (j < (*old)[i+1]) { >>> res->push_back(j); >>> if (j < nelm) { >>> target = (*array)[j]; >>> ++ j; >>> } >>> } >>> } while (j < (*old)[i+1]); >>> } >>> }; >>> } >>> } >>> else { // start with all elements in one segment >>> res->push_back(0); // the first number is always 0 >>> uint32_t j = 1; >>> while (j < nelm) { >>> while (j < nelm && (*array)[j] == target) >>> ++ j; >>> res->push_back(j); >>> if (j < nelm) { >>> target = (*array)[j]; >>> ++ j; >>> } >>> } >>> } >>> if (res->back() < nelm) >>> res->push_back(nelm); >>> return res; >>> } // ibis::colFloats::segment >>> >>> >>> Hope this help. >>> >>> Alexandre Maurel >>> -------------- next part -------------- >>> An HTML attachment was scrubbed... >>> URL: >>> >>> http://hpcrdm.lbl.gov/pipermail/fastbit-users/attachments/20090827/962c3caa/attachment-0001.htm >>> >>> >>> >>> ------------------------------ >>> >>> Message: 2 >>> Date: Thu, 27 Aug 2009 17:59:12 +0200 >>> From: Alexandre Maurel <[email protected] >>> <mailto:[email protected]>> >>> Subject: [FastBit-users] problem in segment method >>> To: [email protected] >>> <mailto:[email protected]> >>> Message-ID: >>> <[email protected] >>> >>> <mailto:[email protected]>> >>> Content-Type: text/plain; charset="iso-8859-1" >>> >>> Hello, >>> >>> I used ardea to generate a table using the following : >>> >>> Barcode Row Col Cycle Sample >>> 100220 1 1 30 sample1 >>> 100220 1 1 36 sample1 >>> 100220 1 2 30 sample1 >>> 100220 1 2 36 sample1 >>> 100220 2 1 30 sample1 >>> 100220 2 1 36 sample1 >>> 100220 2 2 30 sample1 >>> 100220 2 2 36 sample1 >>> 100220 3 3 38 sample2 >>> 100220 4 4 38 sample2 >>> >>> ardea -d tmp -m >>> "Barcode:float,Row:float,Col:float,Cycle:float,Sample:category" -t >>> Array.txt >>> >>> then using ibis, >>> >>> ibis -d tmp -q "select Barcode,Row,Col,Cycle,Sample where Row > 0" >>> >>> result gives .... >>> >>> Barcode, Row, Col, Cycle, Sample (with counts) >>> 100220, 1, 1, 30, sample1, 1 >>> 100220, 1, 1, 36, sample1, 1 >>> 100220, 1, 2, 30, sample1, 1 >>> 100220, 1, 2, 36, sample1, 1 >>> 100220, 2, 1, 30, sample1, 1 >>> 100220, 2, 1, 36, sample1, 1 >>> 100220, 2, 2, 30, sample1, 1 >>> 100220, 2, 2, 36, sample1, 1 >>> 100220, 3, 3, 38, sample2, 2 >>> >>> which is wrong the last row is missing .... >>> >>> Here how I solved this bug >>> >>> array_t<uint32_t>* >>> ibis::colFloats::segment(const array_t<uint32_t>* old) const { >>> array_t<uint32_t>* res = new array_t<uint32_t>; >>> // res->push_back(0); // the first number is always 0 >>> uint32_t j = 0; >>> float target = *(array->begin()); >>> const uint32_t nelm = array->size(); >>> >>> if (old != 0 && old->size()>2) { >>> // find segments with in the previously defined segments >>> for (uint32_t i=0; i<old->size()-1; ++i) { >>> if (j < nelm) { >>> res->push_back(j); >>> target = (*array)[j]; >>> ++ j; >>> if (j < nelm) { >>> do { >>> while (j < (*old)[i+1] && (*array)[j] == >>> target) ++ >>> j; >>> if (j < (*old)[i+1]) { >>> res->push_back(j); >>> if (j < nelm) { >>> target = (*array)[j]; >>> ++ j; >>> } >>> } >>> } while (j < (*old)[i+1]); >>> } >>> }; >>> } >>> } >>> else { // start with all elements in one segment >>> res->push_back(0); // the first number is always 0 >>> uint32_t j = 1; >>> while (j < nelm) { >>> while (j < nelm && (*array)[j] == target) >>> ++ j; >>> res->push_back(j); >>> if (j < nelm) { >>> target = (*array)[j]; >>> ++ j; >>> } >>> } >>> } >>> if (res->back() < nelm) >>> res->push_back(nelm); >>> return res; >>> } // ibis::colFloats::segment >>> >>> >>> Hope this help. >>> >>> Alexandre Maurel >>> -------------- next part -------------- >>> An HTML attachment was scrubbed... >>> URL: >>> >>> http://hpcrdm.lbl.gov/pipermail/fastbit-users/attachments/20090827/0928f095/attachment-0001.htm >>> >>> >>> >>> ------------------------------ >>> >>> Message: 3 >>> Date: Thu, 27 Aug 2009 11:57:05 -0700 >>> From: "K. John Wu" <[email protected] <mailto:[email protected]>> >>> Subject: Re: [FastBit-users] problem in segment method >>> To: FastBit Users <[email protected] >>> <mailto:[email protected]>> >>> Message-ID: <[email protected] >>> <mailto:[email protected]>> >>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >>> >>> Hi, Alexandre, >>> >>> Thanks for bring this issue to our attention. My first >>> impression was >>> that your input data might be missing a end-of-line or some such >>> thing, but guess you suspect that ibis::colFloats::segment is in >>> trouble. Would you mind check whether your input CSV file is indeed >>> missing the end-of-line for the last line? In the mean time, I will >>> look into ibis::colFloats::segment.. >>> >>> John >>> >>> >>> On 8/27/2009 8:59 AM, Alexandre Maurel wrote: >>> > Hello, >>> > >>> > I used ardea to generate a table using the following : >>> > >>> > Barcode Row Col Cycle Sample >>> > 100220 1 1 30 sample1 >>> > 100220 1 1 36 sample1 >>> > 100220 1 2 30 sample1 >>> > 100220 1 2 36 sample1 >>> > 100220 2 1 30 sample1 >>> > 100220 2 1 36 sample1 >>> > 100220 2 2 30 sample1 >>> > 100220 2 2 36 sample1 >>> > 100220 3 3 38 sample2 >>> > 100220 4 4 38 sample2 >>> > >>> > ardea -d tmp -m >>> > "Barcode:float,Row:float,Col:float,Cycle:float,Sample:category" >>> -t Array.txt >>> > >>> > then using ibis, >>> > >>> > ibis -d tmp -q "select Barcode,Row,Col,Cycle,Sample where Row >>> > 0" >>> > >>> > result gives .... >>> > >>> > Barcode, Row, Col, Cycle, Sample (with counts) >>> > 100220, 1, 1, 30, sample1, 1 >>> > 100220, 1, 1, 36, sample1, 1 >>> > 100220, 1, 2, 30, sample1, 1 >>> > 100220, 1, 2, 36, sample1, 1 >>> > 100220, 2, 1, 30, sample1, 1 >>> > 100220, 2, 1, 36, sample1, 1 >>> > 100220, 2, 2, 30, sample1, 1 >>> > 100220, 2, 2, 36, sample1, 1 >>> > 100220, 3, 3, 38, sample2, 2 >>> > >>> > which is wrong the last row is missing .... >>> > >>> > Here how I solved this bug >>> > >>> > array_t<uint32_t>* >>> > ibis::colFloats::segment(const array_t<uint32_t>* old) const { >>> > array_t<uint32_t>* res = new array_t<uint32_t>; >>> > // res->push_back(0); // the first number is always 0 >>> > uint32_t j = 0; >>> > float target = *(array->begin()); >>> > const uint32_t nelm = array->size(); >>> > >>> > if (old != 0 && old->size()>2) { >>> > // find segments with in the previously defined segments >>> > for (uint32_t i=0; i<old->size()-1; ++i) { >>> > if (j < nelm) { >>> > res->push_back(j); >>> > target = (*array)[j]; >>> > ++ j; >>> > if (j < nelm) { >>> > do { >>> > while (j < (*old)[i+1] && (*array)[j] == >>> target) >>> > ++ j; >>> > if (j < (*old)[i+1]) { >>> > res->push_back(j); >>> > if (j < nelm) { >>> > target = (*array)[j]; >>> > ++ j; >>> > } >>> > } >>> > } while (j < (*old)[i+1]); >>> > } >>> > }; >>> > } >>> > } >>> > else { // start with all elements in one segment >>> > res->push_back(0); // the first number is always 0 >>> > uint32_t j = 1; >>> > while (j < nelm) { >>> > while (j < nelm && (*array)[j] == target) >>> > ++ j; >>> > res->push_back(j); >>> > if (j < nelm) { >>> > target = (*array)[j]; >>> > ++ j; >>> > } >>> > } >>> > } >>> > if (res->back() < nelm) >>> > res->push_back(nelm); >>> > return res; >>> > } // ibis::colFloats::segment >>> > >>> > >>> > Hope this help. >>> > >>> > Alexandre Maurel >>> > >>> > >>> > >>> > >>> >>> ------------------------------------------------------------------------ >>> > >>> > _______________________________________________ >>> > FastBit-users mailing list >>> > [email protected] >>> <mailto:[email protected]> >>> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>> >>> >>> ------------------------------ >>> >>> _______________________________________________ >>> FastBit-users mailing list >>> [email protected] <mailto:[email protected]> >>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>> >>> >>> End of FastBit-users Digest, Vol 24, Issue 6 >>> ******************************************** >>> >>> >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> FastBit-users mailing list >>> [email protected] >>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>> >> >> >> ------------------------------ >> >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> >> >> End of FastBit-users Digest, Vol 24, Issue 8 >> ******************************************** >> >> > > > ------------------------------------------------------------------------ > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
