krishna-prasad-s commented on issue #7564:
URL:
https://github.com/apache/arrow-datafusion/issues/7564#issuecomment-1723193554
Sure,
First I open an deltatable (this is from delta-rs with datafusion enabled/)
`let delta_table = delta::open_table_with_storage_options(table_path,
storage_options).await;`
table_path is on ADLS
(`abfss://[email protected]/im/lev/table/`) and
storage options is the data for acess.
One I have this table, I try to use the schema from the table and create a
local table
schema is retrieved as
```
let schema = table.get_schema().unwrap().clone();
```
and another table is created as
```
let response = DeltaTableBuilder::from_uri(table_path)
.with_storage_options(storage_options.unwrap_or_default())
.build();
match response {
Ok(table) => {
let mut config = DeltaOps(table)
.create()
.with_columns(schema.get_fields().clone());
if let Some(name) = name {
config = config.with_table_name(name);
}
let built_table = config.into_future().await?;
return Ok(built_table);
},
Err(err) => return Err(err),
}
```
I now try to get the data frame from the original table
```
let dataframe = self.ctx.read_table(table).unwrap()
```
Here when I inspect the schema it's still returned as `data_type:
Timestamp(Microsecond, None)`
now I tried to get record batches `let record_batches: Vec<RecordBatch> =
cast_df.collect().await.unwrap();` this works but when I see the schema in
record batch , I see this now as `data_type: Timestamp(Nanosecond, None)`.
I even attempted to make a cast when I do the select of the dataframe. `
let cast_expr = c.cast_to( &DataType::Timestamp(TimeUnit::Microsecond, None),
data_schema).unwrap(); `
with something like this on the select statement, but the result is still
the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
