Re: num_level in Parquet Cpp library & how to add a JSON field?

2018-08-19 Thread Uwe L. Korn
Hello Ivy, > Is there any ways to read the data in logical format? because I want to > check if my final output is correct. I usually use the parquet-cli from the parquet-mr project to check if my file is written correctly. This should give you much more informative output. Simple usage: git

Re: num_level in Parquet Cpp library & how to add a JSON field?

2018-08-13 Thread ivywuyzl
Hi Uwe, Thank you for the quick reply! That was very helpful. I have another question regarding your low-level api example here https://github.com/apache/parquet-cpp/blob/master/examples/low-level-api/reader-writer.cc. in the "int32_field", the logical type is TIME_MILLIS and we put dummy

Re: num_level in Parquet Cpp library & how to add a JSON field?

2018-08-03 Thread Uwe L. Korn
Hello Ivy, "primitive binary" means `Type::BYTE_ARRAY`, so you're correct. I have not yet seen anyone use the JSON field with parquet-cpp but the JSON type is simply a binary string with an annotation so I would expect everything to just work. Uwe On Thu, Aug 2, 2018, at 7:59 PM,

num_level in Parquet Cpp library & how to add a JSON field?

2018-08-02 Thread ivywuyzl
Hi, I’m creating a parquet file using the parquet C++ library. I’ve been looking for answers online but still can’t figure out the following questions. 1. What does num_level mean in the WriteBatch method? WriteBatch(int64_t num_levels, const int16_t* def_levels, const