Smotrov commented on issue #10435: URL: https://github.com/apache/datafusion/issues/10435#issuecomment-2103194344
Here is an example file [data.zst.json](https://github.com/apache/datafusion/files/15266143/data.zst.json) And the code, which shows that the file could be perfectly decoded with `async_compression` which is used in DataFusion. Meanwhile it could not be used to read as DataFrame. ```Rust use arrow::datatypes::{Field, Schema}; use datafusion::common::arrow::datatypes::{DataType, TimeUnit}; use datafusion::datasource::file_format::options::NdJsonReadOptions; use datafusion::datasource::file_format::file_compression_type::FileCompressionType; use datafusion::prelude::*; use std::io::Error; use datafusion::error::Result; use async_compression::tokio::bufread::ZstdDecoder; use tokio::io::AsyncReadExt; const FILE_PATH: &str = "data.zst"; #[tokio::main] async fn main() -> Result<(), Error> { // read file with tokio and create a StreamReader let file = tokio::fs::File::open(FILE_PATH).await?; let mut reader = ZstdDecoder::new(tokio::io::BufReader::new(file)); let mut buf = vec![]; reader.read_to_end(&mut buf).await?; println!("📦 Read {} bytes", buf.len()); let schema = Schema::new(vec![ Field::new("OriginalRequest", DataType::Utf8, false), Field::new( "RequestStarted", DataType::Timestamp(TimeUnit::Millisecond, None), false, ), ]); // Create context let ctx = SessionContext::new(); // Read data let json_options = NdJsonReadOptions::default() .file_extension("zst") .file_compression_type(FileCompressionType::ZSTD) .schema(&schema); let df = ctx.read_json(FILE_PATH, json_options).await?; println!("🤨 Hello, ZStd issue!"); df.show_limit(10).await?; Ok(()) } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
