Re: Timestamp coerced by default writing to parquet when resolution is ns (python)

2019-12-06 Thread Weston Pace
Thanks. I similarly noticed that uint32 gets converted to int64. This makes some surface sense as uint32 is a logical type with int64 as the backing physical type. However, uint8, uint16, and uint64 all keep their data types so I was a little surprised. On Fri, Dec 6, 2019 at 6:52 AM Wes McKinn

Re: Timestamp coerced by default writing to parquet when resolution is ns (python)

2019-12-06 Thread Wes McKinney
Some notes * 96-bit nanosecond timestamps are deprecated in the Parquet format by default, so we don't write them by default unless you use the use_deprecated_int96_timestamps flag * 64-bit timestamps are relatively new to the Parquet format, I'm not actually sure what's required to write these. U

Timestamp coerced by default writing to parquet when resolution is ns (python)

2019-12-06 Thread Weston Pace
If my table has timestamp fields with ns resolution and I save the table to parquet format without specifying any timestamp args (default coerce and legacy settings) then it automatically converts my timestamp to us resolution. As best I can tell Parquet supports ns resolution so I would prefer it