yeah, I don't mind looking at the code, but the problem is finding the right code ;) I haven't found any test cases for impala to read/write specific data formats, maybe I will ping the mailing list. Regarding parquet::arrow API, do you have link from github I could chase? I wouldn't mind writing some documentation/examples for the project and make it more approachable for more people :) Many thanks again Wes!
2017-12-09 22:25 GMT+01:00 Wes McKinney <wesmck...@gmail.com>: > I found this comment in Apache Impala helpful, I'm not sure what > better resources are out there outside reading Parquet > implementations: > > https://github.com/apache/impala/blob/master/be/src/ > exec/hdfs-parquet-scanner.h#L80 > > For the parquet::arrow API, you will want to read the header files. > There's some overhead to using the Arrow-based writer API, but I > suspect the overhead is small relative to the other parts of producing > Parquet files. > > - Wes > > On Sat, Dec 9, 2017 at 3:15 PM, Renato Marroquín Mogrovejo > <renatoj.marroq...@gmail.com> wrote: > > Hi Wes, > > > > Thanks a lot for your help! I have been looking at that blog the last > > couple of days but I haven't been able to achieve what I want :( > > Do you know if there is there any actual documentation, test cases or > some > > code I can look at? > > Anyway, this is what I have so far: > > parquet::Int32Writer* int32_writer1 = > > static_cast<parquet::Int32Writer*>(rg_writer->NextColumn()); > > int32_t value = 1; > > value = 1000; > > int16_t definition_level = 2; > > int16_t repetition_level = 0; > > int32_writer1->WriteBatch(1, &definition_level, &repetition_level, > &value); > > > > int16_t rpl = 1; > > int32_writer1->WriteBatch(1, &definition_level, &rpl, &value); > > > > This works better (using the parquet reader doesn't yield into reading > NULL > > values), but I still can't read the resulting parquet file from > > Presto/Athena. > > I would like to have as final result when queries from Presto/Athena: > > id my_array > > 1 array[1000, 1000] > > > > What I currently get is > > id my_array > > 1 > > > > Regarding using parquet::arrow API, is there any docs? that I can look to > > get me started? Also, is there any performance penalties by using > > parquet::arrow instead of the parquet lower api? > > > > 2017-12-09 1:13 GMT+01:00 Wes McKinney <wesmck...@gmail.com>: > > > >> Didn't realize this question was on the Arrow mailing list instead of > >> the Parquet mailing list! > >> > >> You can make things much easier on yourself by putting your data in > >> Arrow arrays and using the parquet::arrow APIs. > >> > >> If you want to write the data using the lower-level Parquet column > >> writer API, you will have to be careful with the repetition/definition > >> levels. In your case, I believe the values you write need to have > >> definition level 2 (the repeated node and optional node both increment > >> the definition level by 1). > >> > >> I find this blog helpful for this > >> https://blog.twitter.com/engineering/en_us/a/2013/ > dremel-made-simple-with- > >> parquet.html. > >> There is also the Google Dremel paper > >> > >> - Wes > >> > >> On Fri, Dec 8, 2017 at 6:19 PM, Renato Marroquín Mogrovejo > >> <renatoj.marroq...@gmail.com> wrote: > >> > Thanks Wes! So I create it this way, but I still don't know how to > >> populate > >> > and > >> > > >> > auto element = PrimitiveNode::Make("element", Repetition::OPTIONAL, > >> > Type::INT32); > >> > auto list = GroupNode::Make("list", Repetition::REPEATED, {element}); > >> > auto my_array = GroupNode::Make("my_array", Repetition::REQUIRED, > {list}, > >> > LogicalType::LIST); > >> > fields.push_back(PrimitiveNode::Make("id", Repetition::REQUIRED, > >> > Type::INT32, LogicalType::NONE)); > >> > fields.push_back(my_array); > >> > auto my_schema = GroupNode::Make("schema", Repetition::REQUIRED, > fields); > >> > > >> > I tried populating it this way: > >> > > >> > parquet::Int32Writer* int32_writer1 = > >> > static_cast<parquet::Int32Writer*>(rg_writer->NextColumn()); > >> > for (int i = 0; i < NROWS_GROUP; i++) { > >> > int32_t value = i; > >> > int16_t definition_level = 1; > >> > int16_t repetition_level = 0; > >> > if ((i+1)%2 == 0) { > >> > repetition_level = 1; // start of a new record > >> > } > >> > int32_writer1->WriteBatch(1, &definition_level, > >> &repetition_level, > >> > &value); > >> > } > >> > > >> > That seems to work, but I can't use the generated file on Athena and > >> using > >> > the parquet_reader from parquet_cpp returns NULLs on the elements. Is > it > >> > that I have to get a handle to the list element? Thanks again for the > >> help! > >> > > >> >
