Hello John, I tried version 1.1.3 it's working. But not now I am confused, as far as I can see the segment methods are the same in both version ...!
Alexandre [email protected] a écrit : > Send FastBit-users mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of FastBit-users digest..." > > > Today's Topics: > > 1. Re: problem in segment method (K. John Wu) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 10 Nov 2009 08:46:38 -0800 > From: "K. John Wu" <[email protected]> > Subject: Re: [FastBit-users] problem in segment method > To: FastBit Users <[email protected]> > Message-ID: <[email protected]> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Hi, Alexandre, > > Would it be possible for you to try version 1.1.3, which was released > on Sept 10 (and newer than the snapshot you used)? > > In the mean time, I will take a more careful look at differences > between your code and what I have now... We are in the middle of > trying to get the next release ready. The current SVN repository > should stabilize in the next few days.. > > John > > > On 11/10/2009 7:41 AM, Alexandre Maurel wrote: > >> Hi John, >> >> I update my code with fastbit snapshot fastbit-scm-2009-09-01. >> I did some test and it is not working for me. >> >> To make it work, I had to go back to the implementation of the segment >> method I first sent. >> >> As attached document you will find the dataset I used for the test. >> >> The dataset contains 152 rows, when I run a query (select <all columns>) >> that should return the complete dataset, 2 rows are missing. >> >> Here are the segment methods implementation I used to make it work. >> >> array_t<uint32_t>* >> ibis::colFloats::segment(const array_t<uint32_t>* old) const { >> array_t<uint32_t>* res = new array_t<uint32_t>; >> // res->push_back(0); // the first number is always 0 >> uint32_t j = 0; >> float target = *(array->begin()); >> const uint32_t nelm = array->size(); >> >> if (old != 0 && old->size()>2) { >> // find segments with in the previously defined segments >> for (uint32_t i=0; i<old->size()-1; ++i) { >> if (j < nelm) { >> res->push_back(j); >> target = (*array)[j]; >> ++ j; >> if (j < nelm) { >> do { >> while (j < (*old)[i+1] && (*array)[j] == >> target) ++ >> j; >> if (j < (*old)[i+1]) { >> res->push_back(j); >> if (j < nelm) { >> target = (*array)[j]; >> ++ j; >> } >> } >> } while (j < (*old)[i+1]); >> } >> }; >> } >> } >> else { // start with all elements in one segment >> res->push_back(0); // the first number is always 0 >> uint32_t j = 1; >> while (j < nelm) { >> while (j < nelm && (*array)[j] == target) >> ++ j; >> res->push_back(j); >> if (j < nelm) { >> target = (*array)[j]; >> ++ j; >> } >> } >> } >> if (res->back() < nelm) >> res->push_back(nelm); >> return res; >> } // ibis::colFloats::segment >> >> and >> >> ibis::array_t<uint32_t>* >> ibis::colUInts::segment(const array_t<uint32_t>* old) const { >> array_t<uint32_t>* res = new array_t<uint32_t>; >> // res->push_back(0); // the first number is always 0 >> uint32_t j = 0; >> uint32_t target = *(array->begin()); >> const uint32_t nelm = array->size(); >> >> if (old != 0 && old->size()>2) { >> // find segments with in the previously defined segments >> for (uint32_t i=0; i<old->size()-1; ++i) { >> if (j < nelm) { >> res->push_back(j); >> target = (*array)[j]; >> ++ j; >> if (j < nelm) { >> do { >> while (j < (*old)[i+1] && (*array)[j] == target) ++ >> j; >> if (j < (*old)[i+1]) { >> res->push_back(j); >> if (j < nelm) { >> target = (*array)[j]; >> ++ j; >> } >> } >> } while (j < (*old)[i+1]); >> } >> }; >> } >> } >> else { // start with all elements in one segment >> res->push_back(0); // the first number is always 0 >> uint32_t j = 1; >> while (j < nelm) { >> while (j < nelm && (*array)[j] == target) >> ++ j; >> res->push_back(j); >> if (j < nelm) { >> target = (*array)[j]; >> ++ j; >> } >> } >> } >> if (res->back() < nelm) >> res->push_back(nelm); >> return res; >> } // ibis::colUInts::segment >> >> Alexandre >> >> [email protected] a ?crit : >> >>> Send FastBit-users mailing list submissions to >>> [email protected] >>> >>> To subscribe or unsubscribe via the World Wide Web, visit >>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>> or, via email, send a message with subject or body 'help' to >>> [email protected] >>> >>> You can reach the person managing the list at >>> [email protected] >>> >>> When replying, please edit your Subject line so it is more specific >>> than "Re: Contents of FastBit-users digest..." >>> >>> >>> Today's Topics: >>> >>> 1. Re: problem in segment method (K. John Wu) >>> >>> >>> ---------------------------------------------------------------------- >>> >>> Message: 1 >>> Date: Fri, 28 Aug 2009 06:56:19 -0700 >>> From: "K. John Wu" <[email protected]> >>> Subject: Re: [FastBit-users] problem in segment method >>> To: FastBit Users <[email protected]> >>> Message-ID: <[email protected]> >>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >>> >>> Hi, Alexandre, >>> >>> You are right, I misunderstood your original message and only realized >>> my mistake after I have sent out the first message. We have adopted >>> your changes to ibis::colFloats::segment and have made changes to >>> other versions of the function segment. >>> >>> Hope you have got a chance to try the latest snapshot >>> <https://codeforge.lbl.gov/snapshots.php?group_id=44>. Please let us >>> know if it correctly resolves the problem for you. >>> >>> Again, thanks for the feedback and your interest in FastBit. >>> >>> John >>> >>> PS: Since we have accepted your modified version of >>> ibis::colFloats::segment, your name is now listed in the list of >>> contributors >>> <https://codeforge.lbl.gov/plugins/scmsvn/viewcvs.php/trunk/AUTHORS?root=fastbit&view=markup> >>> >>> along with a more detailed information in ChangeLog >>> <https://codeforge.lbl.gov/plugins/scmsvn/viewcvs.php/trunk/ChangeLog?root=fastbit&view=markup>. >>> >>> >>> >>> On 8/28/2009 2:13 AM, Alexandre Maurel wrote: >>> >>> >>>> Hello John, >>>> >>>> You suspect that the last row is not in the table because of a >>>> missing end-of-line, but it is not the case. >>>> If you look at the last column of the result set (with counts), the >>>> last row has counts equals to 2. >>>> >>>> Moreover the following query : >>>> ibis -d tmp -q "select Barcode,Row,Col,Cycle,Sample where Row = 4" >>>> return the correct result. >>>> >>>> Barcode, Row, Col, Cycle, Sample (with counts) >>>> 100220, 4, 4, 38, sample2, 1 >>>> >>>> which is the last row of the table. >>>> >>>> Note that the segment method implementation I sent is working >>>> properly on this data set. >>>> >>>> Alexandre >>>> >>>> 2009/8/27 <[email protected] >>>> <mailto:[email protected]>> >>>> >>>> Send FastBit-users mailing list submissions to >>>> [email protected] >>>> <mailto:[email protected]> >>>> >>>> To subscribe or unsubscribe via the World Wide Web, visit >>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>>> or, via email, send a message with subject or body 'help' to >>>> [email protected] >>>> <mailto:[email protected]> >>>> >>>> You can reach the person managing the list at >>>> [email protected] >>>> <mailto:[email protected]> >>>> >>>> When replying, please edit your Subject line so it is more specific >>>> than "Re: Contents of FastBit-users digest..." >>>> >>>> >>>> Today's Topics: >>>> >>>> 1. problem in segment method (Alexandre Maurel) >>>> 2. problem in segment method (Alexandre Maurel) >>>> 3. Re: problem in segment method (K. John Wu) >>>> >>>> >>>> >>>> ---------------------------------------------------------------------- >>>> >>>> Message: 1 >>>> Date: Thu, 27 Aug 2009 17:55:57 +0200 >>>> From: Alexandre Maurel <[email protected] >>>> <mailto:[email protected]>> >>>> Subject: [FastBit-users] problem in segment method >>>> To: [email protected] >>>> <mailto:[email protected]> >>>> Message-ID: >>>> <[email protected] >>>> >>>> <mailto:[email protected]>> >>>> Content-Type: text/plain; charset="iso-8859-1" >>>> >>>> Hello, >>>> >>>> I used ardea to generate a table using the following : >>>> >>>> Barcode Row Col Cycle Sample >>>> 100220 1 1 30 sample1 >>>> 100220 1 1 36 sample1 >>>> 100220 1 2 30 sample1 >>>> 100220 1 2 36 sample1 >>>> 100220 2 1 30 sample1 >>>> 100220 2 1 36 sample1 >>>> 100220 2 2 30 sample1 >>>> 100220 2 2 36 sample1 >>>> 100220 3 3 38 sample2 >>>> 100220 4 4 38 sample2 >>>> >>>> ardea -d tmp -m >>>> "Barcode:float,Row:float,Col:float,Cycle:float,Sample:category" -t >>>> Array.txt >>>> >>>> then using ibis, >>>> >>>> ibis -d tmp -q "select Barcode,Row,Col,Cycle,Sample where Row > 0" >>>> >>>> result gives .... >>>> >>>> Barcode, Row, Col, Cycle, Sample (with counts) >>>> 100220, 1, 1, 30, sample1, 1 >>>> 100220, 1, 1, 36, sample1, 1 >>>> 100220, 1, 2, 30, sample1, 1 >>>> 100220, 1, 2, 36, sample1, 1 >>>> 100220, 2, 1, 30, sample1, 1 >>>> 100220, 2, 1, 36, sample1, 1 >>>> 100220, 2, 2, 30, sample1, 1 >>>> 100220, 2, 2, 36, sample1, 1 >>>> 100220, 3, 3, 38, sample2, 2 >>>> >>>> which is wrong the last row is missing .... >>>> >>>> Here how I solved this bug >>>> >>>> array_t<uint32_t>* >>>> ibis::colFloats::segment(const array_t<uint32_t>* old) const { >>>> array_t<uint32_t>* res = new array_t<uint32_t>; >>>> // res->push_back(0); // the first number is always 0 >>>> uint32_t j = 0; >>>> float target = *(array->begin()); >>>> const uint32_t nelm = array->size(); >>>> >>>> if (old != 0 && old->size()>2) { >>>> // find segments with in the previously defined segments >>>> for (uint32_t i=0; i<old->size()-1; ++i) { >>>> if (j < nelm) { >>>> res->push_back(j); >>>> target = (*array)[j]; >>>> ++ j; >>>> if (j < nelm) { >>>> do { >>>> while (j < (*old)[i+1] && (*array)[j] == >>>> target) ++ >>>> j; >>>> if (j < (*old)[i+1]) { >>>> res->push_back(j); >>>> if (j < nelm) { >>>> target = (*array)[j]; >>>> ++ j; >>>> } >>>> } >>>> } while (j < (*old)[i+1]); >>>> } >>>> }; >>>> } >>>> } >>>> else { // start with all elements in one segment >>>> res->push_back(0); // the first number is always 0 >>>> uint32_t j = 1; >>>> while (j < nelm) { >>>> while (j < nelm && (*array)[j] == target) >>>> ++ j; >>>> res->push_back(j); >>>> if (j < nelm) { >>>> target = (*array)[j]; >>>> ++ j; >>>> } >>>> } >>>> } >>>> if (res->back() < nelm) >>>> res->push_back(nelm); >>>> return res; >>>> } // ibis::colFloats::segment >>>> >>>> >>>> Hope this help. >>>> >>>> Alexandre Maurel >>>> -------------- next part -------------- >>>> An HTML attachment was scrubbed... >>>> URL: >>>> >>>> http://hpcrdm.lbl.gov/pipermail/fastbit-users/attachments/20090827/962c3caa/attachment-0001.htm >>>> >>>> >>>> >>>> ------------------------------ >>>> >>>> Message: 2 >>>> Date: Thu, 27 Aug 2009 17:59:12 +0200 >>>> From: Alexandre Maurel <[email protected] >>>> <mailto:[email protected]>> >>>> Subject: [FastBit-users] problem in segment method >>>> To: [email protected] >>>> <mailto:[email protected]> >>>> Message-ID: >>>> <[email protected] >>>> >>>> <mailto:[email protected]>> >>>> Content-Type: text/plain; charset="iso-8859-1" >>>> >>>> Hello, >>>> >>>> I used ardea to generate a table using the following : >>>> >>>> Barcode Row Col Cycle Sample >>>> 100220 1 1 30 sample1 >>>> 100220 1 1 36 sample1 >>>> 100220 1 2 30 sample1 >>>> 100220 1 2 36 sample1 >>>> 100220 2 1 30 sample1 >>>> 100220 2 1 36 sample1 >>>> 100220 2 2 30 sample1 >>>> 100220 2 2 36 sample1 >>>> 100220 3 3 38 sample2 >>>> 100220 4 4 38 sample2 >>>> >>>> ardea -d tmp -m >>>> "Barcode:float,Row:float,Col:float,Cycle:float,Sample:category" -t >>>> Array.txt >>>> >>>> then using ibis, >>>> >>>> ibis -d tmp -q "select Barcode,Row,Col,Cycle,Sample where Row > 0" >>>> >>>> result gives .... >>>> >>>> Barcode, Row, Col, Cycle, Sample (with counts) >>>> 100220, 1, 1, 30, sample1, 1 >>>> 100220, 1, 1, 36, sample1, 1 >>>> 100220, 1, 2, 30, sample1, 1 >>>> 100220, 1, 2, 36, sample1, 1 >>>> 100220, 2, 1, 30, sample1, 1 >>>> 100220, 2, 1, 36, sample1, 1 >>>> 100220, 2, 2, 30, sample1, 1 >>>> 100220, 2, 2, 36, sample1, 1 >>>> 100220, 3, 3, 38, sample2, 2 >>>> >>>> which is wrong the last row is missing .... >>>> >>>> Here how I solved this bug >>>> >>>> array_t<uint32_t>* >>>> ibis::colFloats::segment(const array_t<uint32_t>* old) const { >>>> array_t<uint32_t>* res = new array_t<uint32_t>; >>>> // res->push_back(0); // the first number is always 0 >>>> uint32_t j = 0; >>>> float target = *(array->begin()); >>>> const uint32_t nelm = array->size(); >>>> >>>> if (old != 0 && old->size()>2) { >>>> // find segments with in the previously defined segments >>>> for (uint32_t i=0; i<old->size()-1; ++i) { >>>> if (j < nelm) { >>>> res->push_back(j); >>>> target = (*array)[j]; >>>> ++ j; >>>> if (j < nelm) { >>>> do { >>>> while (j < (*old)[i+1] && (*array)[j] == >>>> target) ++ >>>> j; >>>> if (j < (*old)[i+1]) { >>>> res->push_back(j); >>>> if (j < nelm) { >>>> target = (*array)[j]; >>>> ++ j; >>>> } >>>> } >>>> } while (j < (*old)[i+1]); >>>> } >>>> }; >>>> } >>>> } >>>> else { // start with all elements in one segment >>>> res->push_back(0); // the first number is always 0 >>>> uint32_t j = 1; >>>> while (j < nelm) { >>>> while (j < nelm && (*array)[j] == target) >>>> ++ j; >>>> res->push_back(j); >>>> if (j < nelm) { >>>> target = (*array)[j]; >>>> ++ j; >>>> } >>>> } >>>> } >>>> if (res->back() < nelm) >>>> res->push_back(nelm); >>>> return res; >>>> } // ibis::colFloats::segment >>>> >>>> >>>> Hope this help. >>>> >>>> Alexandre Maurel >>>> -------------- next part -------------- >>>> An HTML attachment was scrubbed... >>>> URL: >>>> >>>> http://hpcrdm.lbl.gov/pipermail/fastbit-users/attachments/20090827/0928f095/attachment-0001.htm >>>> >>>> >>>> >>>> ------------------------------ >>>> >>>> Message: 3 >>>> Date: Thu, 27 Aug 2009 11:57:05 -0700 >>>> From: "K. John Wu" <[email protected] <mailto:[email protected]>> >>>> Subject: Re: [FastBit-users] problem in segment method >>>> To: FastBit Users <[email protected] >>>> <mailto:[email protected]>> >>>> Message-ID: <[email protected] >>>> <mailto:[email protected]>> >>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >>>> >>>> Hi, Alexandre, >>>> >>>> Thanks for bring this issue to our attention. My first >>>> impression was >>>> that your input data might be missing a end-of-line or some such >>>> thing, but guess you suspect that ibis::colFloats::segment is in >>>> trouble. Would you mind check whether your input CSV file is indeed >>>> missing the end-of-line for the last line? In the mean time, I will >>>> look into ibis::colFloats::segment.. >>>> >>>> John >>>> >>>> >>>> On 8/27/2009 8:59 AM, Alexandre Maurel wrote: >>>> > Hello, >>>> > >>>> > I used ardea to generate a table using the following : >>>> > >>>> > Barcode Row Col Cycle Sample >>>> > 100220 1 1 30 sample1 >>>> > 100220 1 1 36 sample1 >>>> > 100220 1 2 30 sample1 >>>> > 100220 1 2 36 sample1 >>>> > 100220 2 1 30 sample1 >>>> > 100220 2 1 36 sample1 >>>> > 100220 2 2 30 sample1 >>>> > 100220 2 2 36 sample1 >>>> > 100220 3 3 38 sample2 >>>> > 100220 4 4 38 sample2 >>>> > >>>> > ardea -d tmp -m >>>> > "Barcode:float,Row:float,Col:float,Cycle:float,Sample:category" >>>> -t Array.txt >>>> > >>>> > then using ibis, >>>> > >>>> > ibis -d tmp -q "select Barcode,Row,Col,Cycle,Sample where Row >>>> >>>>> 0" >>>>> >>>> > >>>> > result gives .... >>>> > >>>> > Barcode, Row, Col, Cycle, Sample (with counts) >>>> > 100220, 1, 1, 30, sample1, 1 >>>> > 100220, 1, 1, 36, sample1, 1 >>>> > 100220, 1, 2, 30, sample1, 1 >>>> > 100220, 1, 2, 36, sample1, 1 >>>> > 100220, 2, 1, 30, sample1, 1 >>>> > 100220, 2, 1, 36, sample1, 1 >>>> > 100220, 2, 2, 30, sample1, 1 >>>> > 100220, 2, 2, 36, sample1, 1 >>>> > 100220, 3, 3, 38, sample2, 2 >>>> > >>>> > which is wrong the last row is missing .... >>>> > >>>> > Here how I solved this bug >>>> > >>>> > array_t<uint32_t>* >>>> > ibis::colFloats::segment(const array_t<uint32_t>* old) const { >>>> > array_t<uint32_t>* res = new array_t<uint32_t>; >>>> > // res->push_back(0); // the first number is always 0 >>>> > uint32_t j = 0; >>>> > float target = *(array->begin()); >>>> > const uint32_t nelm = array->size(); >>>> > >>>> > if (old != 0 && old->size()>2) { >>>> > // find segments with in the previously defined segments >>>> > for (uint32_t i=0; i<old->size()-1; ++i) { >>>> > if (j < nelm) { >>>> > res->push_back(j); >>>> > target = (*array)[j]; >>>> > ++ j; >>>> > if (j < nelm) { >>>> > do { >>>> > while (j < (*old)[i+1] && (*array)[j] == >>>> target) >>>> > ++ j; >>>> > if (j < (*old)[i+1]) { >>>> > res->push_back(j); >>>> > if (j < nelm) { >>>> > target = (*array)[j]; >>>> > ++ j; >>>> > } >>>> > } >>>> > } while (j < (*old)[i+1]); >>>> > } >>>> > }; >>>> > } >>>> > } >>>> > else { // start with all elements in one segment >>>> > res->push_back(0); // the first number is always 0 >>>> > uint32_t j = 1; >>>> > while (j < nelm) { >>>> > while (j < nelm && (*array)[j] == target) >>>> > ++ j; >>>> > res->push_back(j); >>>> > if (j < nelm) { >>>> > target = (*array)[j]; >>>> > ++ j; >>>> > } >>>> > } >>>> > } >>>> > if (res->back() < nelm) >>>> > res->push_back(nelm); >>>> > return res; >>>> > } // ibis::colFloats::segment >>>> > >>>> > >>>> > Hope this help. >>>> > >>>> > Alexandre Maurel >>>> > >>>> > >>>> > >>>> > >>>> >>>> ------------------------------------------------------------------------ >>>> > >>>> > _______________________________________________ >>>> > FastBit-users mailing list >>>> > [email protected] >>>> <mailto:[email protected]> >>>> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>>> >>>> >>>> ------------------------------ >>>> >>>> _______________________________________________ >>>> FastBit-users mailing list >>>> [email protected] <mailto:[email protected]> >>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>>> >>>> >>>> End of FastBit-users Digest, Vol 24, Issue 6 >>>> ******************************************** >>>> >>>> >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> _______________________________________________ >>>> FastBit-users mailing list >>>> [email protected] >>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>>> >>>> >>> ------------------------------ >>> >>> _______________________________________________ >>> FastBit-users mailing list >>> [email protected] >>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >>> >>> >>> End of FastBit-users Digest, Vol 24, Issue 8 >>> ******************************************** >>> >>> >>> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> > > > ------------------------------ > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > > End of FastBit-users Digest, Vol 27, Issue 9 > ******************************************** > > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
