alamb commented on issue #7564: URL: https://github.com/apache/arrow-datafusion/issues/7564#issuecomment-1728375000
I looked around in the delta.rs code, and while I am not super familiar with how DeltaLake works, it seems like the code in delta.rs treats all timestamps as though it has microsecond precision timestamps: https://github.com/delta-io/delta-rs/blob/a74589be7c39315360925049c716d1d70b906970/rust/src/delta_arrow.rs#L122-L125 Perhaps this issue is related: https://github.com/delta-io/delta/issues/643 > let record_batches: Vec<RecordBatch> = cast_df.collect().await.unwrap(); this works but when I see the schema in record batch , I see this now as data_type: Timestamp(Nanosecond, None). What is the definition of `cast_df`? I see `dataframe defined above, but not `cast_df` ``` let dataframe = self.ctx.read_table(table).unwrap() ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
