Hi, Alexandre,
Would it be possible for you to try version 1.1.3, which was released
on Sept 10 (and newer than the snapshot you used)?
In the mean time, I will take a more careful look at differences
between your code and what I have now... We are in the middle of
trying to get the next release ready. The current SVN repository
should stabilize in the next few days..
John
On 11/10/2009 7:41 AM, Alexandre Maurel wrote:
> Hi John,
>
> I update my code with fastbit snapshot fastbit-scm-2009-09-01.
> I did some test and it is not working for me.
>
> To make it work, I had to go back to the implementation of the segment
> method I first sent.
>
> As attached document you will find the dataset I used for the test.
>
> The dataset contains 152 rows, when I run a query (select <all columns>)
> that should return the complete dataset, 2 rows are missing.
>
> Here are the segment methods implementation I used to make it work.
>
> array_t<uint32_t>*
> ibis::colFloats::segment(const array_t<uint32_t>* old) const {
> array_t<uint32_t>* res = new array_t<uint32_t>;
> // res->push_back(0); // the first number is always 0
> uint32_t j = 0;
> float target = *(array->begin());
> const uint32_t nelm = array->size();
>
> if (old != 0 && old->size()>2) {
> // find segments with in the previously defined segments
> for (uint32_t i=0; i<old->size()-1; ++i) {
> if (j < nelm) {
> res->push_back(j);
> target = (*array)[j];
> ++ j;
> if (j < nelm) {
> do {
> while (j < (*old)[i+1] && (*array)[j] ==
> target) ++
> j;
> if (j < (*old)[i+1]) {
> res->push_back(j);
> if (j < nelm) {
> target = (*array)[j];
> ++ j;
> }
> }
> } while (j < (*old)[i+1]);
> }
> };
> }
> }
> else { // start with all elements in one segment
> res->push_back(0); // the first number is always 0
> uint32_t j = 1;
> while (j < nelm) {
> while (j < nelm && (*array)[j] == target)
> ++ j;
> res->push_back(j);
> if (j < nelm) {
> target = (*array)[j];
> ++ j;
> }
> }
> }
> if (res->back() < nelm)
> res->push_back(nelm);
> return res;
> } // ibis::colFloats::segment
>
> and
>
> ibis::array_t<uint32_t>*
> ibis::colUInts::segment(const array_t<uint32_t>* old) const {
> array_t<uint32_t>* res = new array_t<uint32_t>;
> // res->push_back(0); // the first number is always 0
> uint32_t j = 0;
> uint32_t target = *(array->begin());
> const uint32_t nelm = array->size();
>
> if (old != 0 && old->size()>2) {
> // find segments with in the previously defined segments
> for (uint32_t i=0; i<old->size()-1; ++i) {
> if (j < nelm) {
> res->push_back(j);
> target = (*array)[j];
> ++ j;
> if (j < nelm) {
> do {
> while (j < (*old)[i+1] && (*array)[j] == target) ++
> j;
> if (j < (*old)[i+1]) {
> res->push_back(j);
> if (j < nelm) {
> target = (*array)[j];
> ++ j;
> }
> }
> } while (j < (*old)[i+1]);
> }
> };
> }
> }
> else { // start with all elements in one segment
> res->push_back(0); // the first number is always 0
> uint32_t j = 1;
> while (j < nelm) {
> while (j < nelm && (*array)[j] == target)
> ++ j;
> res->push_back(j);
> if (j < nelm) {
> target = (*array)[j];
> ++ j;
> }
> }
> }
> if (res->back() < nelm)
> res->push_back(nelm);
> return res;
> } // ibis::colUInts::segment
>
> Alexandre
>
> [email protected] a écrit :
>> Send FastBit-users mailing list submissions to
>> [email protected]
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
>> or, via email, send a message with subject or body 'help' to
>> [email protected]
>>
>> You can reach the person managing the list at
>> [email protected]
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of FastBit-users digest..."
>>
>>
>> Today's Topics:
>>
>> 1. Re: problem in segment method (K. John Wu)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Fri, 28 Aug 2009 06:56:19 -0700
>> From: "K. John Wu" <[email protected]>
>> Subject: Re: [FastBit-users] problem in segment method
>> To: FastBit Users <[email protected]>
>> Message-ID: <[email protected]>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>
>> Hi, Alexandre,
>>
>> You are right, I misunderstood your original message and only realized
>> my mistake after I have sent out the first message. We have adopted
>> your changes to ibis::colFloats::segment and have made changes to
>> other versions of the function segment.
>>
>> Hope you have got a chance to try the latest snapshot
>> <https://codeforge.lbl.gov/snapshots.php?group_id=44>. Please let us
>> know if it correctly resolves the problem for you.
>>
>> Again, thanks for the feedback and your interest in FastBit.
>>
>> John
>>
>> PS: Since we have accepted your modified version of
>> ibis::colFloats::segment, your name is now listed in the list of
>> contributors
>> <https://codeforge.lbl.gov/plugins/scmsvn/viewcvs.php/trunk/AUTHORS?root=fastbit&view=markup>
>>
>> along with a more detailed information in ChangeLog
>> <https://codeforge.lbl.gov/plugins/scmsvn/viewcvs.php/trunk/ChangeLog?root=fastbit&view=markup>.
>>
>>
>>
>> On 8/28/2009 2:13 AM, Alexandre Maurel wrote:
>>
>>> Hello John,
>>>
>>> You suspect that the last row is not in the table because of a
>>> missing end-of-line, but it is not the case.
>>> If you look at the last column of the result set (with counts), the
>>> last row has counts equals to 2.
>>>
>>> Moreover the following query :
>>> ibis -d tmp -q "select Barcode,Row,Col,Cycle,Sample where Row = 4"
>>> return the correct result.
>>>
>>> Barcode, Row, Col, Cycle, Sample (with counts)
>>> 100220, 4, 4, 38, sample2, 1
>>>
>>> which is the last row of the table.
>>>
>>> Note that the segment method implementation I sent is working
>>> properly on this data set.
>>>
>>> Alexandre
>>>
>>> 2009/8/27 <[email protected]
>>> <mailto:[email protected]>>
>>>
>>> Send FastBit-users mailing list submissions to
>>> [email protected]
>>> <mailto:[email protected]>
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
>>> or, via email, send a message with subject or body 'help' to
>>> [email protected]
>>> <mailto:[email protected]>
>>>
>>> You can reach the person managing the list at
>>> [email protected]
>>> <mailto:[email protected]>
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of FastBit-users digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>> 1. problem in segment method (Alexandre Maurel)
>>> 2. problem in segment method (Alexandre Maurel)
>>> 3. Re: problem in segment method (K. John Wu)
>>>
>>>
>>>
>>> ----------------------------------------------------------------------
>>>
>>> Message: 1
>>> Date: Thu, 27 Aug 2009 17:55:57 +0200
>>> From: Alexandre Maurel <[email protected]
>>> <mailto:[email protected]>>
>>> Subject: [FastBit-users] problem in segment method
>>> To: [email protected]
>>> <mailto:[email protected]>
>>> Message-ID:
>>> <[email protected]
>>>
>>> <mailto:[email protected]>>
>>> Content-Type: text/plain; charset="iso-8859-1"
>>>
>>> Hello,
>>>
>>> I used ardea to generate a table using the following :
>>>
>>> Barcode Row Col Cycle Sample
>>> 100220 1 1 30 sample1
>>> 100220 1 1 36 sample1
>>> 100220 1 2 30 sample1
>>> 100220 1 2 36 sample1
>>> 100220 2 1 30 sample1
>>> 100220 2 1 36 sample1
>>> 100220 2 2 30 sample1
>>> 100220 2 2 36 sample1
>>> 100220 3 3 38 sample2
>>> 100220 4 4 38 sample2
>>>
>>> ardea -d tmp -m
>>> "Barcode:float,Row:float,Col:float,Cycle:float,Sample:category" -t
>>> Array.txt
>>>
>>> then using ibis,
>>>
>>> ibis -d tmp -q "select Barcode,Row,Col,Cycle,Sample where Row > 0"
>>>
>>> result gives ....
>>>
>>> Barcode, Row, Col, Cycle, Sample (with counts)
>>> 100220, 1, 1, 30, sample1, 1
>>> 100220, 1, 1, 36, sample1, 1
>>> 100220, 1, 2, 30, sample1, 1
>>> 100220, 1, 2, 36, sample1, 1
>>> 100220, 2, 1, 30, sample1, 1
>>> 100220, 2, 1, 36, sample1, 1
>>> 100220, 2, 2, 30, sample1, 1
>>> 100220, 2, 2, 36, sample1, 1
>>> 100220, 3, 3, 38, sample2, 2
>>>
>>> which is wrong the last row is missing ....
>>>
>>> Here how I solved this bug
>>>
>>> array_t<uint32_t>*
>>> ibis::colFloats::segment(const array_t<uint32_t>* old) const {
>>> array_t<uint32_t>* res = new array_t<uint32_t>;
>>> // res->push_back(0); // the first number is always 0
>>> uint32_t j = 0;
>>> float target = *(array->begin());
>>> const uint32_t nelm = array->size();
>>>
>>> if (old != 0 && old->size()>2) {
>>> // find segments with in the previously defined segments
>>> for (uint32_t i=0; i<old->size()-1; ++i) {
>>> if (j < nelm) {
>>> res->push_back(j);
>>> target = (*array)[j];
>>> ++ j;
>>> if (j < nelm) {
>>> do {
>>> while (j < (*old)[i+1] && (*array)[j] ==
>>> target) ++
>>> j;
>>> if (j < (*old)[i+1]) {
>>> res->push_back(j);
>>> if (j < nelm) {
>>> target = (*array)[j];
>>> ++ j;
>>> }
>>> }
>>> } while (j < (*old)[i+1]);
>>> }
>>> };
>>> }
>>> }
>>> else { // start with all elements in one segment
>>> res->push_back(0); // the first number is always 0
>>> uint32_t j = 1;
>>> while (j < nelm) {
>>> while (j < nelm && (*array)[j] == target)
>>> ++ j;
>>> res->push_back(j);
>>> if (j < nelm) {
>>> target = (*array)[j];
>>> ++ j;
>>> }
>>> }
>>> }
>>> if (res->back() < nelm)
>>> res->push_back(nelm);
>>> return res;
>>> } // ibis::colFloats::segment
>>>
>>>
>>> Hope this help.
>>>
>>> Alexandre Maurel
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL:
>>>
>>> http://hpcrdm.lbl.gov/pipermail/fastbit-users/attachments/20090827/962c3caa/attachment-0001.htm
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 2
>>> Date: Thu, 27 Aug 2009 17:59:12 +0200
>>> From: Alexandre Maurel <[email protected]
>>> <mailto:[email protected]>>
>>> Subject: [FastBit-users] problem in segment method
>>> To: [email protected]
>>> <mailto:[email protected]>
>>> Message-ID:
>>> <[email protected]
>>>
>>> <mailto:[email protected]>>
>>> Content-Type: text/plain; charset="iso-8859-1"
>>>
>>> Hello,
>>>
>>> I used ardea to generate a table using the following :
>>>
>>> Barcode Row Col Cycle Sample
>>> 100220 1 1 30 sample1
>>> 100220 1 1 36 sample1
>>> 100220 1 2 30 sample1
>>> 100220 1 2 36 sample1
>>> 100220 2 1 30 sample1
>>> 100220 2 1 36 sample1
>>> 100220 2 2 30 sample1
>>> 100220 2 2 36 sample1
>>> 100220 3 3 38 sample2
>>> 100220 4 4 38 sample2
>>>
>>> ardea -d tmp -m
>>> "Barcode:float,Row:float,Col:float,Cycle:float,Sample:category" -t
>>> Array.txt
>>>
>>> then using ibis,
>>>
>>> ibis -d tmp -q "select Barcode,Row,Col,Cycle,Sample where Row > 0"
>>>
>>> result gives ....
>>>
>>> Barcode, Row, Col, Cycle, Sample (with counts)
>>> 100220, 1, 1, 30, sample1, 1
>>> 100220, 1, 1, 36, sample1, 1
>>> 100220, 1, 2, 30, sample1, 1
>>> 100220, 1, 2, 36, sample1, 1
>>> 100220, 2, 1, 30, sample1, 1
>>> 100220, 2, 1, 36, sample1, 1
>>> 100220, 2, 2, 30, sample1, 1
>>> 100220, 2, 2, 36, sample1, 1
>>> 100220, 3, 3, 38, sample2, 2
>>>
>>> which is wrong the last row is missing ....
>>>
>>> Here how I solved this bug
>>>
>>> array_t<uint32_t>*
>>> ibis::colFloats::segment(const array_t<uint32_t>* old) const {
>>> array_t<uint32_t>* res = new array_t<uint32_t>;
>>> // res->push_back(0); // the first number is always 0
>>> uint32_t j = 0;
>>> float target = *(array->begin());
>>> const uint32_t nelm = array->size();
>>>
>>> if (old != 0 && old->size()>2) {
>>> // find segments with in the previously defined segments
>>> for (uint32_t i=0; i<old->size()-1; ++i) {
>>> if (j < nelm) {
>>> res->push_back(j);
>>> target = (*array)[j];
>>> ++ j;
>>> if (j < nelm) {
>>> do {
>>> while (j < (*old)[i+1] && (*array)[j] ==
>>> target) ++
>>> j;
>>> if (j < (*old)[i+1]) {
>>> res->push_back(j);
>>> if (j < nelm) {
>>> target = (*array)[j];
>>> ++ j;
>>> }
>>> }
>>> } while (j < (*old)[i+1]);
>>> }
>>> };
>>> }
>>> }
>>> else { // start with all elements in one segment
>>> res->push_back(0); // the first number is always 0
>>> uint32_t j = 1;
>>> while (j < nelm) {
>>> while (j < nelm && (*array)[j] == target)
>>> ++ j;
>>> res->push_back(j);
>>> if (j < nelm) {
>>> target = (*array)[j];
>>> ++ j;
>>> }
>>> }
>>> }
>>> if (res->back() < nelm)
>>> res->push_back(nelm);
>>> return res;
>>> } // ibis::colFloats::segment
>>>
>>>
>>> Hope this help.
>>>
>>> Alexandre Maurel
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL:
>>>
>>> http://hpcrdm.lbl.gov/pipermail/fastbit-users/attachments/20090827/0928f095/attachment-0001.htm
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 3
>>> Date: Thu, 27 Aug 2009 11:57:05 -0700
>>> From: "K. John Wu" <[email protected] <mailto:[email protected]>>
>>> Subject: Re: [FastBit-users] problem in segment method
>>> To: FastBit Users <[email protected]
>>> <mailto:[email protected]>>
>>> Message-ID: <[email protected]
>>> <mailto:[email protected]>>
>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>
>>> Hi, Alexandre,
>>>
>>> Thanks for bring this issue to our attention. My first
>>> impression was
>>> that your input data might be missing a end-of-line or some such
>>> thing, but guess you suspect that ibis::colFloats::segment is in
>>> trouble. Would you mind check whether your input CSV file is indeed
>>> missing the end-of-line for the last line? In the mean time, I will
>>> look into ibis::colFloats::segment..
>>>
>>> John
>>>
>>>
>>> On 8/27/2009 8:59 AM, Alexandre Maurel wrote:
>>> > Hello,
>>> >
>>> > I used ardea to generate a table using the following :
>>> >
>>> > Barcode Row Col Cycle Sample
>>> > 100220 1 1 30 sample1
>>> > 100220 1 1 36 sample1
>>> > 100220 1 2 30 sample1
>>> > 100220 1 2 36 sample1
>>> > 100220 2 1 30 sample1
>>> > 100220 2 1 36 sample1
>>> > 100220 2 2 30 sample1
>>> > 100220 2 2 36 sample1
>>> > 100220 3 3 38 sample2
>>> > 100220 4 4 38 sample2
>>> >
>>> > ardea -d tmp -m
>>> > "Barcode:float,Row:float,Col:float,Cycle:float,Sample:category"
>>> -t Array.txt
>>> >
>>> > then using ibis,
>>> >
>>> > ibis -d tmp -q "select Barcode,Row,Col,Cycle,Sample where Row
>>> > 0"
>>> >
>>> > result gives ....
>>> >
>>> > Barcode, Row, Col, Cycle, Sample (with counts)
>>> > 100220, 1, 1, 30, sample1, 1
>>> > 100220, 1, 1, 36, sample1, 1
>>> > 100220, 1, 2, 30, sample1, 1
>>> > 100220, 1, 2, 36, sample1, 1
>>> > 100220, 2, 1, 30, sample1, 1
>>> > 100220, 2, 1, 36, sample1, 1
>>> > 100220, 2, 2, 30, sample1, 1
>>> > 100220, 2, 2, 36, sample1, 1
>>> > 100220, 3, 3, 38, sample2, 2
>>> >
>>> > which is wrong the last row is missing ....
>>> >
>>> > Here how I solved this bug
>>> >
>>> > array_t<uint32_t>*
>>> > ibis::colFloats::segment(const array_t<uint32_t>* old) const {
>>> > array_t<uint32_t>* res = new array_t<uint32_t>;
>>> > // res->push_back(0); // the first number is always 0
>>> > uint32_t j = 0;
>>> > float target = *(array->begin());
>>> > const uint32_t nelm = array->size();
>>> >
>>> > if (old != 0 && old->size()>2) {
>>> > // find segments with in the previously defined segments
>>> > for (uint32_t i=0; i<old->size()-1; ++i) {
>>> > if (j < nelm) {
>>> > res->push_back(j);
>>> > target = (*array)[j];
>>> > ++ j;
>>> > if (j < nelm) {
>>> > do {
>>> > while (j < (*old)[i+1] && (*array)[j] ==
>>> target)
>>> > ++ j;
>>> > if (j < (*old)[i+1]) {
>>> > res->push_back(j);
>>> > if (j < nelm) {
>>> > target = (*array)[j];
>>> > ++ j;
>>> > }
>>> > }
>>> > } while (j < (*old)[i+1]);
>>> > }
>>> > };
>>> > }
>>> > }
>>> > else { // start with all elements in one segment
>>> > res->push_back(0); // the first number is always 0
>>> > uint32_t j = 1;
>>> > while (j < nelm) {
>>> > while (j < nelm && (*array)[j] == target)
>>> > ++ j;
>>> > res->push_back(j);
>>> > if (j < nelm) {
>>> > target = (*array)[j];
>>> > ++ j;
>>> > }
>>> > }
>>> > }
>>> > if (res->back() < nelm)
>>> > res->push_back(nelm);
>>> > return res;
>>> > } // ibis::colFloats::segment
>>> >
>>> >
>>> > Hope this help.
>>> >
>>> > Alexandre Maurel
>>> >
>>> >
>>> >
>>> >
>>>
>>> ------------------------------------------------------------------------
>>> >
>>> > _______________________________________________
>>> > FastBit-users mailing list
>>> > [email protected]
>>> <mailto:[email protected]>
>>> > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
>>>
>>>
>>> ------------------------------
>>>
>>> _______________________________________________
>>> FastBit-users mailing list
>>> [email protected] <mailto:[email protected]>
>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
>>>
>>>
>>> End of FastBit-users Digest, Vol 24, Issue 6
>>> ********************************************
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> _______________________________________________
>>> FastBit-users mailing list
>>> [email protected]
>>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
>>>
>>
>>
>> ------------------------------
>>
>> _______________________________________________
>> FastBit-users mailing list
>> [email protected]
>> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
>>
>>
>> End of FastBit-users Digest, Vol 24, Issue 8
>> ********************************************
>>
>>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> FastBit-users mailing list
> [email protected]
> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
_______________________________________________
FastBit-users mailing list
[email protected]
https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
