Nevermind.. Found the problem.. https://issues.apache.org/jira/browse/DRILL-4203
David Lee Vice President | BlackRock Phone: +1.415.670.2744 | Mobile: +1.415.706.6874 From: Lee, David Sent: Tuesday, November 01, 2016 11:21 AM To: '[email protected]' <[email protected]> Subject: Parquet Date Format Problem I created a parquet file using Drill, but date values in the parquet files don’t appear to be a logical INT32 type and as such when I’m trying to read the parquet file in Spark it looks corrupted.. Here’s my test case.. A. Create a test.txt file in /tmp: as_of 2016-09-30 B. Convert it to parquet using Drill: 0: jdbc:drill:zk=local> create table dfs.tmp.`/test` as select cast(as_of AS date) as as_of from table(dfs.`/tmp/test.txt`(type => 'text', fieldDelimiter => ',', extractHeader => true)); C. Read the new file using Drill which looks fine: 0: jdbc:drill:zk=local> select * from dfs.`/tmp/test`; +-------------+ | as_of | +-------------+ | 2016-09-30 | +-------------+ D. However running parquet-tools on it gives a completely different result: java -jar parquet-tools-1.6.1-SNAPSHOT.jar head -n3 /tmp/test as_of = 4898250 java -jar parquet-tools-1.6.1-SNAPSHOT.jar schema /tmp/test/0_0_0.parquet message root { required int32 as_of (DATE); } According to the Parquet docs.. 4898250 days after Jan 1st 1970 is sometime in the year 15,435.. https://github.com/Parquet/parquet-format/blob/master/LogicalTypes.md DATE DATE is used to for a logical date type, without a time of day. It must annotate an int32 that stores the number of days from the Unix epoch, 1 January 1970. David Lee Vice President | BlackRock Phone: +1.415.670.2744 | Mobile: +1.415.706.6874 This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See http://www.blackrock.com/corporate/en-us/compliance/email-disclaimers for further information. Please refer to http://www.blackrock.com/corporate/en-us/compliance/privacy-policy for more information about BlackRock’s Privacy Policy. For a list of BlackRock's office addresses worldwide, see http://www.blackrock.com/corporate/en-us/about-us/contacts-locations. © 2016 BlackRock, Inc. All rights reserved.
