Hi, Stanislav, Thanks for the information. We will add a note to the documentation of ardea.cpp.
John On 3/6/13 12:32 PM, Stanislav P. Seltser wrote: > i run into the issue too and its frustrated me for a while > this occurs if file is format is dos based as opposed to unix based. > convert the file to unix and then everything will be parsed correctly. > this is an assumption in ardea parser that everything is unix based. > > ---------------------------------------------------------------------- > *From: *[email protected] > *To: *[email protected] > *Sent: *Wednesday, March 6, 2013 3:00:56 PM > *Subject: *FastBit-users Digest, Vol 67, Issue 1 > > Send FastBit-users mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of FastBit-users digest..." > > > Today's Topics: > > 1. TPCH tests for fastbit (amihay gonen) > 2. Re: TPCH tests for fastbit (K. John Wu) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 6 Mar 2013 00:51:24 +0200 > From: amihay gonen <[email protected]> > Subject: [FastBit-users] TPCH tests for fastbit > To: [email protected] > Message-ID: > <CAKb+SBWcQ=gMaOFqyOcNUg2rGTAsZyBr3xKcV8Nk=rpgotw...@mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Hi , > I'm trying to make environment for testing TPCH queries on fastbit (if > anyone has those queries "translated" to ibis query format it will be > great). > > I've started by trying to convert row data (1G rows) to col data using > ardea tool > > On the big table lineitem , the row data is about 780M and the col > data got > 1.8G ,this was strange , by looking into the directory where the col > files > are i see the following : > > -rw-rw-r--. 1 agonen agonen 211M Mar 6 00:07 comment > -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 commitdate > -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 discount > -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 linenumber > -rw-rw-r--. 1 agonen agonen 104M Mar 6 00:07 linestatus > -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 orderkey > -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 partkey > -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 price > -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 quantity > -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 receipdate > -rw-rw-r--. 1 agonen agonen 958M Mar 6 00:07 returnflag <--- too big for > varchar(1) > -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 shipdate > -rw-rw-r--. 1 agonen agonen 256M Mar 6 00:07 shipinstuct > -rw-rw-r--. 1 agonen agonen 202M Mar 6 00:07 shipmode > -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 suppkey > -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 tax > -rw-rw-r--. 1 agonen agonen 1.6K Mar 6 00:07 -part.txt > > the returnflag ,for some reason contain the all record. > > I'm using the fastbit-ibis1.3.5 and the command to convet the csv to > col is > : > > ~/Code/fastbit-ibis1.3.5/examples/ardea -d > ~/Code/fastbit_col_data/lineitem.tbl -b \| -m > "orderkey:int,partkey:int,suppkey:int,linenumber:int,quantity: > float,price:float,discount:float,tax:float,returnflag:text,linestatus:key,shipdate:int,commitdate:int,receipdate:int,shipinstuct:text,shipmode:key,comment:text" > -t ~/Code/rowdata/lineitem.tbl > > > the base file look like that: > > 1|155190|7706|1|17|21168.23|0.04|0.02|N|O|1996-03-13|1996-02-12|1996-03-22|DELIVER > IN PERSON|TRUCK|egular courts above the| > 1|67310|7311|2|36|45983.16|0.09|0.06|N|O|1996-04-12|1996-02-28|1996-04-20|TAKE > BACK RETURN|MAIL|ly final dependencies: slyly bold | > 1|63700|3701|3|8|13309.60|0.10|0.02|N|O|1996-01-29|1996-03-05|1996-01-31|TAKE > BACK RETURN|REG AIR|riously. regular, express dep| > > > any idea ? > > thanks amihay > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://hpcrdm.lbl.gov/pipermail/fastbit-users/attachments/20130306/b23ba158/attachment-0001.htm > > > ------------------------------ > > Message: 2 > Date: Tue, 05 Mar 2013 22:30:49 -0800 > From: "K. John Wu" <[email protected]> > Subject: Re: [FastBit-users] TPCH tests for fastbit > To: FastBit Users <[email protected]> > Cc: amihay gonen <[email protected]> > Message-ID: <[email protected]> > Content-Type: text/plain; charset=ISO-8859-1 > > Hi, Amihay, > > I just took the three rows you included in the message and tried it > with your ardea command line. Things seem to have completed > successfully in my macbook. In these three rows, the value of > returnflag are all 'N'. Not sure what platform you are using.. > > Maybe your run of ardea has encountered some errors. If you have > captured the print out from this command line, would you mind share it > with me? > > Not sure what you plan to do with the data files, but one thing you > should know is that there are many queries from TPCH that can not be > handled by FastBit. For example, FastBit does not deal with date as > nicely as DBMS. FastBit does not do multi-table joins neither. > > John > > > On 3/5/13 2:51 PM, amihay gonen wrote: >> Hi , >> I'm trying to make environment for testing TPCH queries on fastbit (if >> anyone has those queries "translated" to ibis query format it will be >> great). >> >> I've started by trying to convert row data (1G rows) to col data using >> ardea tool >> >> On the big table lineitem , the row data is about 780M and the col >> data got 1.8G ,this was strange , by looking into the directory where >> the col files are i see the following : >> >> -rw-rw-r--. 1 agonen agonen 211M Mar 6 00:07 comment >> -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 commitdate >> -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 discount >> -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 linenumber >> -rw-rw-r--. 1 agonen agonen 104M Mar 6 00:07 linestatus >> -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 orderkey >> -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 partkey >> -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 price >> -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 quantity >> -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 receipdate >> -rw-rw-r--. 1 agonen agonen 958M Mar 6 00:07 returnflag <--- too big >> for varchar(1) >> -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 shipdate >> -rw-rw-r--. 1 agonen agonen 256M Mar 6 00:07 shipinstuct >> -rw-rw-r--. 1 agonen agonen 202M Mar 6 00:07 shipmode >> -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 suppkey >> -rw-rw-r--. 1 agonen agonen 69M Mar 6 00:07 tax >> -rw-rw-r--. 1 agonen agonen 1.6K Mar 6 00:07 -part.txt >> >> the returnflag ,for some reason contain the all record. >> >> I'm using the fastbit-ibis1.3.5 and the command to convet the csv to >> col is : >> >> ~/Code/fastbit-ibis1.3.5/examples/ardea -d >> ~/Code/fastbit_col_data/lineitem.tbl -b \| -m >> "orderkey:int,partkey:int,suppkey:int,linenumber:int,quantity: >> > float,price:float,discount:float,tax:float,returnflag:text,linestatus:key,shipdate:int,commitdate:int,receipdate:int,shipinstuct:text,shipmode:key,comment:text" >> -t ~/Code/rowdata/lineitem.tbl >> >> >> the base file look like that: >> >> > 1|155190|7706|1|17|21168.23|0.04|0.02|N|O|1996-03-13|1996-02-12|1996-03-22|DELIVER >> IN PERSON|TRUCK|egular courts above the| >> > 1|67310|7311|2|36|45983.16|0.09|0.06|N|O|1996-04-12|1996-02-28|1996-04-20|TAKE >> BACK RETURN|MAIL|ly final dependencies: slyly bold | >> > 1|63700|3701|3|8|13309.60|0.10|0.02|N|O|1996-01-29|1996-03-05|1996-01-31|TAKE >> BACK RETURN|REG AIR|riously. regular, express dep| >> >> >> any idea ? >> >> thanks amihay >> >> >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users >> > > > ------------------------------ > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > > > End of FastBit-users Digest, Vol 67, Issue 1 > ******************************************** > > > > _______________________________________________ > FastBit-users mailing list > [email protected] > https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users > _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
