Hello,
The type of my_primary_key is a long. I was able to reproduce the error
without the join, I also noticed that it does not hit the seg fault if the
category column is omitted. The following program will hit the error.
$ cat first_data_file.csv
1,93.19,AAA
2,49.14,BBB
3,50.41,CCC
4,58.59,AAA
5,19.53,CCC
$ cat second_data_file.csv
3,49.19,DDD
4,59.10,EEE
5,34.48,FFF
6,91.49,AAA
7,19.50,BBB
$ cat loading_error.cc
#include <memory>
#include <ibis.h>
int main(int argc, char **argv)
{
char existing_dir[] = "existing_dir";
char first_incoming_dir[] = "first_incoming_dir";
char second_incoming_dir[] = "second_incoming_dir";
std::auto_ptr<ibis::tablex> firstTable(ibis::tablex::create());
firstTable->addColumn("my_primary_key", ibis::LONG);
firstTable->addColumn("my_double_value", ibis::DOUBLE);
firstTable->addColumn("my_category_value", ibis::CATEGORY);
firstTable->readCSV("first_data_file.csv", 0, first_incoming_dir, ",");
firstTable->write(first_incoming_dir, "working", NULL, NULL, NULL);
firstTable->clearData();
ibis::part existing_part(existing_dir);
existing_part.append(first_incoming_dir);
existing_part.commit(first_incoming_dir);
existing_part.purgeIndexFiles();
existing_part.buildIndexes();
existing_part.emptyCache();
std::auto_ptr<ibis::tablex> secondTable(ibis::tablex::create());
secondTable->addColumn("my_primary_key", ibis::LONG);
secondTable->addColumn("my_double_value", ibis::DOUBLE);
secondTable->addColumn("my_category_value", ibis::CATEGORY);
secondTable->readCSV("second_data_file.csv", 0, second_incoming_dir,
",");
secondTable->write(second_incoming_dir, "working", NULL, NULL, NULL);
secondTable->clearData();
ibis::part second_part(second_incoming_dir);
existing_part.deactivate("my_primary_key = 1");
existing_part.purgeInactive();
existing_part.append(second_incoming_dir);
}
Thank you John,
Greg
On Sun, Aug 12, 2012 at 3:27 PM, K. John Wu <[email protected]> wrote:
> Hi, Greg,
>
> Thanks for the information. Looks like we might have neglected to
> close some index files or somehow mishandled some index files. There
> is only easy thing for us to check, this is related to the handling of
> categorical values (the columns of type ibis::CATEGORY). Would you
> mind tell us if my_primary_key is an integer column or a CATEGORY column?
>
> If it is not a CATEGORY, then we might have something a little bit
> more complex. We would appreciate a small test case to replicate the
> problem.
>
> John
>
>
> On 8/10/12 5:32 PM, Greg Barker wrote:
> > Hello -
> >
> > I am attempting to append some new data to some existing data, and ran
> > into some trouble. When loading, I join the new data to the existing
> > data on a particular column, and then deactivate & purgeInactive on
> > the matching records. Then when I try to append the new data to the
> > existing data, I hit a seg fault using rev 536. If I
> > call purgeIndexFiles before the append, it seems to avoid the crash,
> > but I wasn't sure if that was recommended?
> >
> > My code is essentially:
> >
> > ibis::part existing_part("my_data");
> > ibis::part incoming_part("new_data");
> > std::auto_ptr<ibis::quaere>
> > join(ibis::quaere::create(&existing_part, &incoming_part,
> > "my_primary_key"));
> > std::auto_ptr<ibis::table> rs(join->select("my_primary_key"));
> > //then build the where clause
> > working_part.deactivate("my_primary_key in (3, 4, 5)");
> > working_part.purgeInactive();
> > working_part.append(incoming_data);
> >
> >
> > Which yields the following:
> >
> > part[my_data]::deactivate marked 9 rows as inactive, leaving 10
> > active rows out of 19
> > part[my_data]::purgeInactive to remove 9 out of 19 rows
> > Warning -- fileManager::flushDir can not remove in-memory file
> > (my_data/my_primary_key.idx). It is in use
> > Warning -- fileManager::flushDir(my_data) finished with 1 file
> > still in memory
> > Constructed a part named my_data
> > filter::sift1S -- processing data partition my_data
> > Segmentation fault (core dumped)
> >
> > Many Thanks,
> > Greg
>
_______________________________________________
FastBit-users mailing list
[email protected]
https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
