ajazam opened a new issue, #13602:
URL: https://github.com/apache/datafusion/issues/13602
### Describe the bug
I am trying to create a parquet file with hive partitioning, from csv data
and get error
Error: External(NotImplemented("it is not yet supported to write to hive
partitions with datatype Float64"))
### To Reproduce
_**main.rs**_
use std::fs::File;
use std::io::Write;
use arrow::datatypes::{DataType, Field, Schema};
use datafusion::prelude::*;
use tempfile::tempdir;
#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
let dir = tempdir()?;
let file_path = dir.path().join("example.csv");
let mut file = File::create(&file_path)?;
file.write_all(
r#"dte,ot
2016-07-01 00:00:00,2
2016-07-01 06:45:00,3"#
.as_bytes())?;
let file_path = file_path.to_str().unwrap();
let ctx = SessionContext::new();
let csv_df = ctx.read_csv(file_path, CsvReadOptions::default()).await?;
csv_df.show().await?;
let schema = Schema::new(vec![
Field::new("dte",
DataType::Timestamp(arrow::datatypes::TimeUnit::Second, None), false),
Field::new("ot", DataType::UInt16, false),
]);
ctx.register_csv("data" ,file_path,
CsvReadOptions::new().schema(&schema).has_header(true)).await?;
let df = ctx.sql("copy (SELECT dte, ot, EXTRACT(YEAR FROM dte) AS year
from data) to './partitioned_output' stored as parquet PARTITIONED BY
(year)").await?;
df.count().await?;
Ok(())
}
_**cargo.toml**_
[package]
name = "datafusion_csv"
version = "0.1.0"
edition = "2021"
[dependencies]
tokio = { version = "1", features = ["full"] }
datafusion = "43.0.0"
arrow = "53.3.0"
tempfile = "3.14.0"
### Expected behavior
I am expecting a folder year=2016 containing a parquet file
### Additional context
I was original trying to have folders for month and day, couldn't get the
application to work and then created this simpler example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
