Hi Stephen, I'm not sure I follow your scenario. So you have your own Thrift generator; you then write (outside of Hive, using your own input/output format) the Thrift objects out into HDFS using your own custom Parquet output format. You have two questiosn: 1. Is there anything special you need to tell Hive to make these parquet files readable, other than that they are Parquet? 2. How does one deal with thrift evolution?
Assuming I have this right, here are some answers: 1. There should not be; "stored as parquet" looks at the schema stored in parquet. If you have errors, it may be that (a) there are bugs in how your custom OF writes parquet metadata, or (b) there are bugs in Hive's parquet reader -- please do share if you find any. 2. Schema evolution is rough. At some point, the pig loader used names instead of ids, Pig would break from renaming a field without changing a type. Not sure how Hive reader is implemented, someone else on this list might know. I think it's actually pretty static, since all the column names, etc, get written into the metastore by hive. So you'd have to refresh table definitions even if you simply add a new column, much less change an old one. On Mon, Dec 7, 2015 at 8:12 AM, Stephen Bly <[email protected]> wrote: > Greetings Parquet experts. I am in need of a little help. > > I am a (very) Junior developer at my company, and I have been tasked with > adding the Parquet file format to our Hadoop ecosystem. Our main use case > is in creating Hive tables on Parquet data and querying them. > > As you know, Hive can create Parquet tables with the STORED AS PARQUET > command. However, we use a custom Thrift generator to generate Scala code > (similar to Twitter’s Scrooge, if you’re familiar with that). Thus, I am > not sure if the above command will work. > > I tested it out and it is hit and miss -- I'm able to create tables, but > often get errors when querying them that I am still investigating. > > Hive allows you to be more custom by using INPUT FORMAT ... OUTPUT FORMAT > ... ROW FORMAT SERDE .... We already have a custom Parquet input and output > format. I am wondering if I will need to create a custom serde. I am not > really sure where to start for this. > > Furthermore, I am worried about changing schema, i.e. if we change our > Thrift definitions and then try to read data that was written in the old > format. This is because Hive uses the name to select a column,not Thrift > IDs which never change. Do I need to point to the Thrift definition from > inside the Parquet file to keep track of a changing schema? I feel like > this does not make sense as Parquet is self-describing, storing its own > schema in metadata. > > Any help would be appreciated. I have read a ton of documentation but > nothing seems to address my (very specific) question. Thank you in advance.
